The Talent Evaluation: an evaluation of the tri-agencies' support for research training and talent development

Final Report

Evaluation Division of the Social Sciences and Humanities and Natural Sciences and Engineering Research Councils
Prairie Research Associates
Réseau Circum Inc
Evaluation Unit of the Canadian Institutes of Health Research

Acknowledgement

We would like to thank the more than 17,000 current and former graduate students who contributed to this study. Your time and effort to describe your experience through interview or by survey has been vital and is deeply appreciated. We would also like to thank participating researchers who hold agency grants and who supervise students on their research teams, as well as the individuals representing stakeholder organizations who provided input, including other funders of graduate students and organizations representing students and universities. This study benefitted greatly from individuals at organizations who were conducting their own studies and who graciously shared what they were learning. Thank you for your time and your commitment to improving how the three federal research funding agencies support graduate student training.

The evaluation team included three embedded graduate students. Two other graduate students contributed at important timepoints. Having a graduate student perspective on data collection, analysis and interpretation was critical. We wish to further acknowledge that most of the members of the evaluation team at the agencies were graduate students within the timeframe covered by this study (2014-21), and some of us were also beneficiaries of agency support during that time.

Finally, we want to recognize the members of the Evaluation Advisory Committee and Evaluation Working Group, and the leadership at SSHRC, NSERC and CIHR, for their vision, guidance and support.

Table of Contents

• Executive Summary
• Introduction
• Purpose of the evaluation
• Scope of the evaluation
• Evaluation questions and approach to the study
• About the Talent portfolio
• Overview of findings
• Recommendations
• References
• Appendix A. List of Acronyms
• Appendix B. Glossary of key terms
• Appendix C. Portfolio funding opportunities
• Appendix D. Methodology

Executive Summary

This report provides a summary of key findings from an evaluation of the support provided by the three federal research funding agencies for graduate student training. This multiyear, horizontal study was designed to inform strategic decisions about the funding for research training provided by the Social Sciences and Humanities Research Council (SSHRC), the Natural Sciences and Engineering Research Council (NSERC) and the Canadian Institutes of Health Research (CIHR). Findings fall under three priority areas: student access to agency support; student training experience; and student career trajectory. Impacts of the COVID-19 pandemic are also documented.

The evaluation’s strategic purpose is reflected in its recommendations. These are outlined in the table below and detailed in the report. The recommendations focus on opportunities for the agencies to diversify their reach to graduate students and to renew their contribution to research training. The recommendations also recognize the shared role and responsibility among multiple actors for the development of Canada’s future research community and for building a more coherent and equitable student funding system.

This report is organized in four parts, beginning with an introduction, followed by a description of the agencies’ funding portfolio for graduate student training. Section three provides an overview of key findings, followed by detailed recommendations in section four. Additional descriptions of agency programs and an overview of methodology are provided in the appendix.

For readers interested in additional detail about this study and its findings, three companion reports are available containing in-depth findings. A companion report is available for each of the three priority focus areas for this evaluation: Student Access, Training Experience and Trajectory. These reports contain contextual background, descriptive data and detailed results to support a fulsome understanding of the findings. These companion reports are available upon request from evaluation@sshrc-crsh.gc.ca.

Introduction

This evaluation assessed the tri- agencies’^{Footnote 2} funding for graduate student training. Specifically, the evaluation studied how the agencies’ funding enables student access to training opportunities, contributes to a rich and inclusive training environment, and influences student trajectory. In doing so, the evaluation considered opportunities to strengthen the agencies’ collective efforts to advance research training. This was a horizontal study of the agencies’ investment in student training. Its intended purpose, scope and approach are outlined below.

First, it is essential to highlight that the journey of students through their graduate training is complex and shaped by countless factors and considerations. In 2019-20, of the more than 1.3 million students enrolled in universities in Canada, over 273,000 were graduate students. Of that number, 79% were enrolled at the master’s level and 21% were enrolled in doctoral studies (PSIS, 2021). These individuals were pursuing their training in close to 80 different universities that vary dramatically in size, location, language and range of programs offered. These students’ experience formed the backdrop of this evaluation.

The path leading to graduate studies is unique to each student. Social, cultural and economic factors undeniably influence how a student’s studies align with their learning vision and career goals. Pressures on students to conform to the learning models offered by universities, and for universities to adapt to the diversity among students, are continuously at play, creating tensions that are reshaping graduate learning in Canada. These forces are co-evolving in an ecosystem fuelled by several factors including its decentralized institutional structure and the multiplicity of actors and stakeholders involved. Within this context, the importance of the tri-agencies’ funding is growing alongside increases in graduate student enrollment, in the diversity of students and variety of career paths, as well as growth in interdisciplinary and international collaboration and in the active roles played by funders (Statistics Canada, 2020b).

Canadian universities rely predominantly on government funding and tuition fees to operate. In Canada, the provinces are responsible for education. However, recognizing the importance of education to national priorities, such as the economy, federal departments and agencies have also provided support to postsecondary institutions and students. Federal programs are intended to achieve objectives consistent with the mandate of the department or agency.^{Footnote 3}

In 2018-19, provincial governments contributed 35% (or $11 billion) of total university funding. Tuition fees contributed 29% (or $9 billion) and the federal government contributed 11% (or $3 billion). Additional funding came from donors, industry and not-for-profit organizations (Statistics Canada, 2020a). As a major component of the federal contribution, the agencies provided an estimated $1.46 billion to universities in 2018-19 through the funding opportunities studied as part of the scope of this evaluation, or almost half the federal contribution in that year. Other funding from the agencies and other federal departments and agencies made up the remainder. The various funders attach conditions to the support they provide which, when added to the vision and priorities of each university, ultimately shape the experience of students.

In considering the evaluation’s findings, it is critical to keep this broader context in mind. The programs offered by the three agencies remain one of many inputs that shape graduate studies in Canada. They interact with other sources of funding, are implemented through processes unique to each institution, and take different shapes in different places. Their impact is further determined by the circumstance of each student enrolled in graduate studies. This means that it is not possible to account for all the interrelated factors of a student's academic experience. Ultimately, the findings reported below are shaped by myriads of individual student stories.

Purpose of the evaluation

This evaluation was undertaken to support strategic decisions about talent-related funding at the agencies and to identify opportunities to strengthen the agencies’ efforts to support research training. A primary purpose was to inform the development of a new Tri-agency Training Strategy. The evaluation also meets Treasury Board requirements related to evaluation.^{Footnote 4} The evaluation was further designed as a first phase of a longitudinal study to assess trends influencing student training and students’ long-term career trajectories.^{Footnote 5}

The vision for the Tri-agency Training Strategy is to support, inspire and empower the next generation of leaders, within and beyond the research ecosystem, for the benefit of Canada. The strategy aims to be trainee-centric, evidence-based and transparent, while communicating a shared vision among the three agencies and upholding the principles of equity, diversity and inclusion (CRCC, 2022).

The agencies have a long-standing commitment to evidence-based program development. Leveraging evaluation to support strategy is especially important in complex environments where success depends on innovation, adaptation and continuous learning. This evaluation not only provides evidence to support strategy development, it also supports the other principles behind the training strategy, including transparency through public reporting of the evaluation’s results and adoption of a student-centred approach and EDI lens in the study design.

Scope of the evaluation

This evaluation focused on the agencies’ funding for graduate student training (i.e., training for master’s and doctoral students).^{Footnote 6} The agencies’ funding portfolio for training includes two broad streams:

• Direct stream: direct support to students through scholarship and fellowship^{Footnote 7} awards.
• Indirect stream: indirect support for training through grants to researchers. Researchers are expected to provide experiential training for students by involving them in the implementation of their agency-funded research or through other modes of research-related training. Researchers may also provide financial support to students in the form of stipends or salaries.

Each funding stream contains multiple funding opportunities (FOs).^{Footnote 8} Information about the overall size and composition of the portfolio is provided below. Summary descriptions of the individual FOs in the portfolio are available in Appendix C. It is worth highlighting that academic institutions and the researchers within them contribute to delivering all the funding in the portfolio, in both streams. They participate in the selection of recipients, enable the flow of funding to students, and deliver the training experiences for which this funding is intended.

This marks the first time that the agencies’ funding for graduate students has been examined collectively. The high-level perspective has allowed for unique insights about agency investment in research training and how this investment interacts in the broader funding ecosystem.

Evaluation questions and approach to the study

As defined in the Treasury Board Secretariat Policy on Results (TBS, 2016), evaluations typically focus on programs, policies and priorities. They also examine other units or themes as determined by user needs. Three priority areas were identified for this evaluation in consultation with agency staff and representatives from stakeholder organizations: student access, training experience and trajectory. These priority areas were the basis for the following evaluation questions:

1. Who does the portfolio reach? This question focused on student access to funding in the portfolio. It considered whether the agencies have the right mix of FOs to reach a diverse array of graduate students, considering the agencies’ EDI commitment to access for all trainees.
2. How does this portfolio contribute to the research training environment? This question focused on student training experience and the agencies’ influence on the research training environment.
3. Does this funding influence student career trajectories? This question focused on how students’ career goals and early-career outcomes relate to their funding and training opportunities.
4. What should the agencies consider in the context of changes in the training environment and/or student experience due to COVID-19? The impact of the pandemic on the training environment was also considered as part of this evaluation.

The first three questions speak to the relevance and effectiveness of the agencies’ funding for student training. As the evaluation took place during the COVID-19 pandemic, the fourth question was used to document the impacts being experienced by the research and graduate student community, as well as potential near-term implications for consideration by the agencies. The evaluation applied an EDI lens throughout all stages of the study.^{Footnote 9} The EDI lens informed decisions about the evaluation's priority areas of focus, its questions and indicators, data sources^{Footnote 10} and analytical approach. It ensured explicit attention to diversity and inclusion and potential barriers and enablers for underrepresented groups.^{Footnote 11} Additional detail on the approach to EDI is provided in Appendix D.

As noted above, the evaluation was designed to be student-centered. This principle was integrated in a variety of ways, including through up-front consultations with student organizations when determining priority focus areas, the direct participation of graduate students on the evaluation team, selecting study methods to ensure depth and breadth of student perspectives and allocating resources accordingly, as well as a primary analytical focus at the level of student outcomes (relative to potential benefits at other levels, for example, for research labs or institutions).

A draft theory of change was developed for the portfolio in the first phase of the evaluation. It was used to clarify the portfolio’s intended reach, the distinctions between roles and responsibilities of the agencies relative to those of academic institutions and researchers, and proposed pathways to intended impact. Agency logic models and other descriptions of activities and intended outcomes are specific to each individual program. The theory of change was therefore helpful to elevate focus to a strategic and portfolio-level view.^{Footnote 12}

The evaluation used a mixed methods design, incorporating six lines of evidence implemented in three phases. Primary data were collected through key informant interviews, case studies, focus groups and surveys. The evaluation also made extensive use of secondary data, including agency administrative data, external stakeholder reports and survey data from external organizations.

Important methodological considerations

When interpreting the findings reported below, there are some key considerations to keep in mind. First, as is common in real-world social interventions, the variation in student experience can make outcomes of agency funding challenging to trace. This includes, for example, variation in the implementation related to these programs within academic institutions, such as differences in the way that training is delivered by student supervisors and grant holders. This results in variation in outcomes among students within the same program/FO categories. This variation can mean that small differences between groups may go undetected. Related to this, students can participate in multiple agency and non-agency programs. Interactions among interventions may exist that are unknown. Mitigating this challenge was the primary motivation behind the mixed method multiphase study design, to allow for increased sensitivity to contextual factors in primary data collection; fielding the student survey to a large respondent sample; and multiple lines of evidence to enable triangulation.

Second, samples for the student survey were limited in important ways:

1. the sample for unsuccessful doctoral awards applicants was more inclusive of early institutional stage review steps for SSHRC than for NSERC and CIHR, due to differences in available administrative data at time of sampling. Because of this difference between the agencies, additional subgroup analyses were conducted postsurvey to validate results;
2. a two-stage approach to sampling for indirectly-support students was necessary because the agencies do not routinely collect contact information for students receiving support through all research grants. This required the sample for students in the indirect stream to be developed through an initial survey of grant holders. To reduce grant holder respondent burden, the indirect sample was limited to current and very recently graduated students: this evaluation was designed to be part of a future longitudinal study on indirect stream student trajectory. Further, some grant holders may have provided contact information for a subset of their students (i.e., not all the students funded by the grant) for a variety of reasons: this is a potential source of sampling bias.^{Footnote 13} Analysis of nonresponse bias^{Footnote 14} could be conducted only at grant and grant-holder level, as student-level population data were not available for the indirect stream. This limited the study’s ability to use inferential statistics in the analysis of indirect stream outcomes: this was taken into consideration in drawing conclusions and developing recommendations. Qualifiers are included in the reporting of results, and caution is recommended for the reader in generalizing findings beyond the indirect stream sample.
3. With respect to student survey results overall, it is important to draw attention to the large size of the sample because, with large samples, some results may be statistically significant but not necessarily meaningful. For this reason, qualitative results, descriptive statistics and contextual information are provided to support interpretation of significant results. These need to be given due attention.

Third, data collection for this study was conducted in 2021 and winter/spring 2022. Student training experiences in 2020 and 2021 were disrupted by the COVID-19 pandemic, and the influence of the pandemic on student training is likely to be felt for some time forward. Additional cohort analyses were incorporated where pre-pandemic participant data were available and where pandemic impact was anticipated to assess for differences in results between pandemic-period and prepandemic participants.

Lastly, while an EDI lens was applied throughout the study, the ability to incorporate intersectional analysis was limited for a number of reasons, described in Appendix D.

About the Talent portfolio

The agencies invest in the Talent portfolio to achieve research training outcomes―for students and for Canada’s future research capacity. The funding is intended to enhance research training and is ultimately about the research that will be done in Canada a decade or more from now.

The funding opportunities (FOs) in the portfolio have potential to work in concert to provide a range of training opportunities for students with a variety of backgrounds and research interests. In this way the portfolio may contribute to greater diversity in the future research ecosystem. The FOs are also intended to work together to contribute to a rich and inclusive training experience for students, and meet overarching aims for attracting, retaining and developing research talent in Canada.

As noted above, the agencies provide support for graduate student training through two broad streams of funding: direct and indirect.

Direct stream

The agencies’ scholarship and fellowship awards are intended to reduce a student’s need to take paid employment and thereby increase their ability to focus on study. Ultimately, award holders are expected to be more likely to complete their degrees and at a faster pace than peers. Direct funding awards are also understood to carry additional benefits such as recognition and/or prestige and eligibility to access supplemental agency funding.^{Footnote 15} In addition, having funding independent of a supervisor or institution is thought to allow students more independence to seek out mentors and training environments that best fit their interests. Direct awards are offered at different levels (Table 1).

The CGS M awards are allocated to academic institutions and each institution performs all aspects of the selection process for its own awards. The CGS D and agency-specific doctoral awards follow a common application process and are adjudicated at two levels for most applicants, first within institutions and later in a national-level competition managed by the agencies. Institutions are provided a quota, or upper limit, of applications they can submit for inclusion in the national-level doctoral competition.

The agency-specific doctoral awards reflect additional investment by each agency in direct funding for students. They include features not available in the CGS D, such as an option to hold the award at an institution outside Canada, and slight variations related to the context of each research domain.

To distinguish the direct stream funding from forms of support through the indirect stream, the agencies’ direct stream funding is referred to in this report as “Type 1” support.

Indirect stream

The indirect stream is made up of grants to researchers that include student training and/or mentorship objectives.^{Footnote 21} In many cases, they include training among their selection criteria and state an explicit expectation that grantees will involve students in their research and/or use grant funds for training, helping students gain research experience and skills. A portion of the funds may be used by researchers to pay salaries or stipends to students. In this way the funding may provide some financial support to students in addition to experiential learning. The indirect stream funding has three types:

1. Grants for research by individual researchers or teams of researchers (Type 2).^{Footnote 22} Participation of students is expected to advance their research skills and understanding of the research process. These grants account for 93% of financial investment in the indirect stream of the portfolio and 96% of active grants/year.^{Footnote 23}
2. Grants for research collaborations between multiple organizations (Type 3). The participation of students is expected to provide exposure to collaborative and often cross-sectoral research activities, leading to skills in working with partners. They account for about 5% of investment and 4% of active grants/year.^{Footnote 24}
3. Grants for development of programs or approaches to enrich student training (Type 4). These expand on modes of training in diverse ways, such as internships with non-academic organizations, and are intended to improve students’ job readiness and transition to the workforce. They account for about 2% of annual investment and fewer than 1% of active grants/year.

At a high level, the four types of funding in the portfolio reflect a set of tools deployed by the agencies for research training. To the extent that they complement each other, they can enable greater access for students and a fuller spectrum of training opportunities. In this respect, the portfolio’s composition reflects an existing strategy in action at the agencies (Patton & Patrizi, 2009) that has developed through a process of experimentation and learning over time (Mintzberg, 2007).^{Footnote 25}

Agency investment through the portfolio

The agencies provide a total of about $1.56 billion per year to universities through the two streams. A portion of research grant (indirect stream) funds are allocated to graduate students and the proportion allocated varies by agency (see Figure 1, below). Unlike indirect funding, 100% of the portfolio’s direct stream funds are allocated to graduate students. The total amount of financial support provided to graduate students through the two streams averaged just under $460 million per year over 2018-21.

Description of figure

Figure 1 shows a doughnut chart of tri-agency annual average investments in the Talent portfolio between 2018-21 for direct and indirect stream support to graduate students. For SSHRC, direct stream investment was $100.9 million; $44.5 million was allocated to graduate students through the indirect stream. NSERC invested $68.8 million in the direct stream and $159.7 million was allocated to graduate students through the indirect stream. CIHR invested $26.5 million in the direct stream and $58.5 million was allocated to graduate students through the indirect stream.

Direct stream investment. The three agencies invested an average of $196.1 million per year over the three fiscal years 2018-21 in the direct stream. Direct stream investment has increased from $177.7 million in 2013-14 to $212.75 million in 2020-21, which is reflected in an increase in the number of awards. It is important to note that while the amount invested and the number of awards have increased since 2014, levels in 2021 are similar to what they were in 2010, as shown in Figure 2, below. Meanwhile, the domestic graduate student population has grown; the most recent population estimate indicates an increase of 16% at master’s level and a 1% at doctoral level between 2010 and 2019 (PSIS, 2021). The size of the domestic master’s student population has grown at a faster rate than doctoral students. The number of awards has not kept pace with this increase in population.

Description of figure

Figure 2 shows a bar graph of the total number of award recipients each year from 2009 to 2021, including CGS and agency-specific awards. In 2009, the number of recipients was 5,176; in 2021 it was 5,483. Between 2011 and 2018, the number of recipients dropped to below 5,000, ranging from 4,070 (the lowest in 2012) to 4,532 (the highest in 2013). In 2019, the number increased again to above 5,000 recipients.

In 2020-21, the agencies were providing financial support to 9,708 graduate students through active awards in the direct stream.^{Footnote 27} This represents about 5% of Canada’s domestic graduate student population, including 2% of master’s and 17% of doctoral students.^{Footnote 28} Due to the higher dollar value and longer duration of the doctoral awards, three-quarters (75%) of direct stream annual investment is in doctoral-level awards and one-quarter (25%) is in master’s-level awards.^{Footnote 29}

Most of the agencies’ direct awards are available only to Canadian citizens, permanent residents and protected persons.^{Footnote 30} However, it is worth noting that there has been substantial growth in the international student population in Canada at both degree levels, in addition to the increases in domestic student population noted above. In particular, Canada’s doctoral student growth after 2009 is almost entirely in international students. International students do receive support through the agencies’ indirect stream, described next.

Indirect stream investment. Total investment in the indirect stream of the portfolio was $1.51 billion in 2020-21. Note that this is total grant funding. The portions allocated to student salaries/stipends are reported below. In 2020-21, the indirect stream funded 25,395 active research grants through which participating students received experiential training. These grants were held by 33,789 individual researchers, including principal and co-investigators.

The indirect stream is larger than the direct stream in terms of overall dollar investment and the number of students receiving financial support, except in social sciences and humanities (SSH). Of total indirect funding within the scope of the evaluation, SSHRC indirect grants account for 10%, NSERC for 32% and CIHR for 58%.^{Footnote 31} For SSHRC, the estimated number of students receiving indirect financial support is comparable to the direct stream.^{Footnote 32}

The percentage of funds allocated by grant holders to graduate students varies by research domain. On average, about one-third of grant funds from SSHRC and NSERC is used to pay salaries or stipends to graduate students. The proportion of CIHR grant funds is smaller, at 3% on average, shown in Figure 3.^{Footnote 33} These differences relate, at least in part, to differences in the needs of research projects funded under these FOs. It is important to recognize that the primary objective of indirect stream grants is to fund research, with graduate student training included under that primary objective. Decisions about the amount of grant funding to be allocated to graduate student financial support are made entirely outside the agencies, except for NSERC’s Type 4 training grants.^{Footnote 34}

Description of figure

Figure 3 shows three doughnut charts, one for each agency. The charts show indirectly funded salary support allocated to student and non-student groups by agency between 2018-21. For SSHRC, 35% of indirect funds in this portfolio were allocated by grant holders to non-salary expenses, 32% to graduate students, 23% to other highly qualified personnel, 6% to postdoctoral fellows, and 4% to undergraduate students. For NSERC, 45% of funds were allocated by grant holders to non-salary expenses, 37% to graduate students, 8% to other highly qualified personnel, 6% to postdoctoral fellows, and 4% to undergraduate students. For CIHR, 80% of its funds were allocated by grant holders to non-salary expenses, 14% to other highly qualified personnel, 3% to graduate students, 2% to postdoctoral fellows, and 1% to undergraduate students.

The number of students involved in each research project varies among grants. However, on average, each grant in the indirect stream provided financial support to three graduate students in 2020-21. The agencies’ indirect funding reaches approximately 30% of the graduate student population (including both domestic and international students).^{Footnote 35} Details about student involvement are provided in the companion report on student experience.^{Footnote 36}

The amount of financial support received by each student through indirect stream grants averaged $5,704 per academic semester (median $5,000), as reported by students on the survey for this evaluation.^{Footnote 37} However, stipends vary widely in amount and in duration: this variation is a key characteristic of the agencies’ indirect support to students. Depending on decisions at the level of the institution and grant holder, indirectly funded students may receive funds in a variety of forms, including as a stipend, a salary as a research assistant (RAship), or as a nonsalary research fellowship.^{Footnote 38} The variation in stipend amounts relates, in part, to the flexibility available to grant holders in the use of this funding. For students who receive funding through RAships as pay for work performed, it may relate to the extent of their participation in the project (hours worked). The agencies do not control or offer guidance on form, amounts or duration of funding to individual students.^{Footnote 39} For simplicity, this report uses stipend as an umbrella term for support to indirectly funded students.

Overview of findings

The following section summarizes the evaluation’s key findings. It is important to note that findings related to CGS D and agency-specific doctoral award recipients are reported separately in this report only when the findings differ between these recipient groups. This is for simplicity and to avoid repetition in the text. Because the CGS D and agency-specific doctoral awards follow a common process, findings in this report related to the application and review process are applicable to all doctoral awards. A few exceptions are noted explicitly in the text, for example, related to differences in degree-stage eligibility requirements. With respect to outcomes, findings are consistent between CG D and agency-specific doctoral recipients except where specified below.^{Footnote 40}

Who does the portfolio of talent-related funding reach?

The agencies mostly reach students who are enrolled full time in a research-intensive degree program (98%). About 75% of participating students are enrolled at large universities.^{Footnote 41} Overall, an estimated two-thirds (67%) of students participating in the portfolio are doctoral students. In these respects, the agencies’ reach aligns with objectives to contribute to the supply of trained researchers to the academic sector and increase Canada’s research capacity. However, this means that the agencies reach very few students who have not already committed to a research path. Overall, in terms of numbers of students participating and of overall financial investment, the portfolio is oriented to students who are further along in what has been called the pipeline of talent for research-intensive institutions. ⁴² The agencies’ reach to master’s students, who are earlier in their academic path, is limited.^{Footnote 43}

The agencies’ reach to part-time students is also very limited, at about 3% of students in the portfolio. For context, estimates of part-time enrollment in the overall graduate student population range from 13% (CGPSS, 2019) to 23% (PSIS, 2019, 2021). Part-time enrollment tends to be higher for some student subgroups, including students with dependents and women (CGPSS, 2019). These findings have implications for the agencies’ broader EDI objectives: this is revisited below under recommendations.

In other respects, participation in the portfolio reflects much of the diversity among Canada’s graduate students. Some equity-deserving groups are represented at levels comparable to or above their estimated representation in the graduate student population. This includes students who identify as women, with gender identities other than man or woman, with LGBTQ2+ communities, as well as students who identify as having disabilities. This is also the case for francophone students.^{Footnote 44}

Similar or greater representation relative to population does not imply absence of barriers for students within those groups, however. Students with disabilities and students identifying with LGBTQ2+ communities, for example, were more likely to report barriers to direct awards compared with other groups. The evaluation results also indicate a number of areas where multiple factors converge in ways that can affect a student’s access to the portfolio. Limited reach to part-time students, in both streams, is one example. The alignment in reach of the two streams to full-time students may disproportionately affect some groups more likely to study part time and for whom part-time studies may be necessary, for example, some students with disabilities, students with dependents and students facing greater financial constraint.

Representation of some student subgroups is somewhat lower than expected in some funding types in the portfolio. Overall, the evaluation found lower representation in the direct stream than in the indirect stream for students identifying with visible minority communities,^{Footnote 45} students with dependents, as well as first-generation students.

The data indicate an increase in visible minority representation in direct awards over time, yet the data show a slight underrepresentation remains. That is due―in part―to the application rate. This is consistent with data from interviews suggesting that students who are members of visible minority communities may be less likely to apply for agency awards.

Representation of students with dependents is also generally lower among direct stream award recipients, and lowest among students who have received both types of funding (indirect and direct). The survey data are consistent with data from interviews, suggesting that students with dependents may be less successful in awards competitions and may be less likely to participate in research projects outside their thesis. A variety of reasons were reported, including having less time to develop strong networks inside academia, which is a factor linked to access to both direct and indirect stream support.^{Footnote 46}

For students who identify as Indigenous, representation in SSHRC funding (3%) and CIHR funding (2%) is equal to the population estimates for graduate students in SSH-related and health-related research-based programs, respectively. For NSERC’s funding, while the proportion of students self-identifying as Indigenous (1%) is lower than the population estimate in NSE (2%), the small difference may be due to multiple factors including, for example, variation in self-identification or disciplinary variation in survey response for this evaluation and/or external surveys used to estimate overall population.

In 2019, the agencies introduced an initiative to allow academic institutions to submit CGS D applications by students who self-identify as Indigenous above an institution’s annual CGS D quota. The data indicate a corresponding increase in representation of Indigenous students in competition year 2019 for all three agencies.^{Footnote 47} However, the increase was sustained post-2019 for CIHR only. The data suggest that Indigenous student participation in some of CIHR’s indirect stream funding may have contributed to their higher representation among CIHR direct award holders. This is an example of potential synergy between the two streams of funding, in which indirect stream participation may contribute to direct stream access.

Although they are not eligible for most direct stream funding from the agencies, the agencies reach international students through indirect funding. Among students receiving financial support through SSHRC and CIHR indirect grants, about one-quarter are international students (21% and 25%, respectively), which is about proportionate to estimated population. The proportion is higher than estimated population for NSERC: 45% of NSERC’s indirectly funded recipients are international students.

As noted above, representation of students identifying with some equity-deserving groups is equal to or higher in the indirect stream than in the agencies’ direct awards programs (for example, members of visible minority groups, Indigenous students in health), but is lower for other groups (for example, women, members of LGBTQ2+ communities and students with disabilities, as well as francophone students). The data suggest that lower representation of some groups in the indirect stream may be due, in part, to relatively lower representation of these groups among international students. For example, representation of students who self-identify as having a disability is 7-10 percentage points lower among participating international students than domestic students.

Findings from the evaluation point to the importance of access to information and to support within institutions for students to successfully access both direct and indirect stream support^{Footnote 48}. Emphasis on prior academic history in the awards selection processes is a key factor affecting access to direct funding: GPA, in particular, was identified by grant holders, institutional staff and students as the primary determinant of access to awards. Agency efforts to reduce emphasis on GPA, as part of EDI efforts, have had limited effectiveness in practice. GPA remains a primary factor for multiple reasons, including the highly competitive nature of the agencies’ awards combined with administrative constraints at all levels to review applications, as well as the decentralized and multilevel selection process for the awards, which introduces local-level variations in approach.^{Footnote 49} Of note, the agencies removed first-class academic standing as an eligibility criterion for the CGS D awards beginning in the 2019 competition. However, among survey respondents who had applied to the CGS D or agency-specific doctoral awards after the change was implemented, only 39% of respondents indicated awareness that a first-class average GPA was no longer an eligibility requirement. The first-class academic standing requirement remains in place for the CGS M, although exceptions are permitted. Overall, the data reflect structural issues in the system that make it challenging to successfully integrate more flexibility and breadth in selection criteria for direct awards.

A particular area of concern raised by students is the transparency and perceived fairness of the direct funding selection process. Almost one-third of recipients and half of unsuccessful applicants disagreed that the process was transparent. Fewer than half of award recipients agreed that the selection process was fair. Transparency concerns were also raised in the previous evaluation of the CGS (CIHR, 2016).

Analysis of open comments to the student survey indicate that the lack of clarity for students about the process contributes to a perception of subjectivity in awards selection, concerns among students that the institutional-level selection process allows for bias, and that perceptions of bias (whether warranted or not) can lead to tensions in research units and labs. This is a critical focus area for the agencies if they want to foster trust in the funding system.

With respect to transparency of the selection process, students reported being unclear about who reviews their application and at what levels within the academic institution, what information reviewers use or do not use, and how the information is used. Concerns included whether institutional-level reviewers have access to information they provide as special considerations on the application, as this can include very personal and sensitive information. Concerns also included being unsure whether reviewers use information that is not on the application but that they may know about the student, as reviewers may be in the same department or faculty as the student. A few respondents reported omitting information on their application (e.g., about a disability) because they were unsure of who within their department or institution would have access to the information. Of note, concerns about data privacy and fear of discrimination have been identified as factors influencing willingness to self-identify in other studies (e.g., Grimes et al., 2020; Morina, 2022; OCHRO, 2022; Shpigelman et al., 2022). This is also a critical focus area for the agencies.

In addition to information about the review process, more useful feedback on unsuccessful applications was highlighted by students as important to increase transparency and support learning, especially for students who are new to research-intensive studies or who have less access to mentorship and other supports inside academia.

Differential access to information among students was identified consistently across lines of evidence. In interviews, students who face the greatest challenges navigating the academic system were understood to face greater challenges accessing both streams of support, for example, first-generation students, students with dependents and other students who may have weaker relationships inside academia. This was reported to affect awareness of opportunities to participate in agency-funded research projects and likelihood to apply for awards. Case study participants and respondents to the student survey described challenges in navigating the available information on how to apply for direct awards and reliance on a variety of informal sources, such as peers and other contacts inside academia.

A lack of access to such supports was described as a barrier by respondents to the survey. Respondents to the student survey who reported experiencing barriers to accessing agency direct awards were invited to describe them. From a sample of 1,200 comments,^{Footnote 50} the largest category (43% of comments) referenced the awards application process as a barrier. The process to apply for awards was described as involving a steep learning curve for first-time applicants and requiring substantial time and effort to navigate and complete. More than one-third of respondents in this group described the need for support from contacts inside academia to navigate the process, and their own lack of such supports as a barrier.

The findings point to the critical role played by academic supervisors to guide students toward funding and funded training opportunities in both streams. Indirect stream access, in particular, is largely through supervisor-student relationships and these relationships are the primary enabler to accessing indirect stream support. Most students receiving indirect financial support from NSERC (95%) and CIHR (85%) receive the support of their academic supervisor or co-supervisor. In contrast, only half (53%) of students receiving SSHRC indirect funding do so through their academic supervisor or co-supervisor (Figure 4). Students whose academic supervisors do not hold agency grants have less access to agency indirect funding, particularly in NSE and in health.

Description of figure

Figure 4 shows three pie charts, one for each agency. The pie charts show proportions of students participating in indirectly funded research projects in 2020-21 by their relationship to the grant holder. For 53% of students involved in SSHRC-funded research, the grant holder was their academic supervisor or co-supervisor. For 14% of students, the grant holder had another type of supervisory relationship with the student. For 33% of students, the grant holder had no supervisory relationship with the student. In the case of NSERC-funded research, the grant holder was the academic supervisor or co-supervisor for 95% of students. For 2% of students, the grant holder had another type of supervisory relationship with the student. For 3% of students, the grant holder had no supervisory relationship with the student. In the case of CIHR-funded research, for 85% of students the grant holder was their academic supervisor or co-supervisor. For 6% of students, the grant holder had another type of supervisory relationship with the student. For 8% of students, the grant holder had no supervisory relationship with the student.

In interviews, grant holders described focusing on multiple considerations when selecting students to supervise and become part of their research team, reflecting the variety of priorities, needs and contexts of their research projects. Apart from the time to participate, as noted above, students’ fit with supervisors’ research interests, rather than GPA, was identified as the most important factor. Grant-holder flexibility to select students to join their research teams may allow a researcher to actively recruit from underrepresented groups within their departments and faculties, including from among undergraduates, as reported by some grant holders. Overall, agency grant holders reported having access to a large pool of highly qualified student applications to choose from, and that agency funding for their projects adds capacity to develop their research teams. Some grant holders noted that their grants also support their departments’ priorities related to student recruitment, and that their ability to attract graduate students to their institution (especially at the doctoral level) is closely linked to their capacity to support these students through agency funding.

The factors influencing access are multiple and interacting, which means that there is no single or simple action for the agencies to take. It also means that an action to address one issue can exacerbate another. As one example, any action to increase access to agency direct awards that also serves to make eligibility criteria or the application process more complicated could impact students with less access to supports and mentorship within institutions to navigate that process successfully. As another example, any actions that increase the administrative burden within institutions are likely to reinforce reliance on metrics such as GPA as a way to for institutions to efficiently manage applications. Either would risk undermining the original intent of the agencies. Opportunities do exist for the agencies to diversify and expand their reach, summarized in the recommendations below.

How does this portfolio contribute to the research training environment?

The agencies’ investment through this funding portfolio contributes to the richness and variety of opportunities for research training and to research capacity within academic institutions. The impact on training experiences for individual students varies, however, and strengths and weaknesses are evident among specific areas of training for students in both streams.

Enriched training environment and expanded research capacity. The indirect stream provides funding for research projects and programs that generate experiential opportunities for students. In 2020-21, the indirect stream funded 25,395 active research grants, held by 33,789 researchers across a broad spectrum of research topics across SSH, NSE, and health. More than 75,000 graduate students, or 30% of the population, are estimated to receive some level of financial support from these grants annually. Almost all indirect stream students responding to the survey for this evaluation (91%) confirmed that they were actively participating in the agency-funded research project, in addition to receiving financial support. Students’ active participation (over and above financial support) is fundamental to the experiential learning the indirect FOs are intended to enable: this strong response is an important validation.

With respect to the direct stream’s contribution to the training system, the doctoral awards provide funding at levels greater than the average reported stipend received through agency research grants. The CGS M does not. Award and stipend levels are revisited below.

The direct stream investment also has some carryover effect to non-recipient students and to overall research capacity. This is because, as a common practice in institutions, other funding (including agency indirect funding) is redirected away from agency-award recipients to students who do not hold agency awards. The agencies’ awards therefore free up other funds to allow a greater number of existing students to receive at least some financial support and to allow for recruitment of new students into graduate programs. As graduate students are contributing members of research teams, this helps expand current research capacity. It is important to note that reallocation of funding can impact individual award recipients. This is revisited below.

Student training experience outcomes. The evaluation asked students in case study interviews and the student survey about their opportunities to develop skills in multiple areas important for research-intensive careers. Their responses were compared with data from interviews with academic supervisors and staff in academic institutions.^{Footnote 51} An overview follows.

The evaluation found some evidence that the agencies’ direct awards increase opportunities for skills development and research-related experience. Where statistically significant differences were found in survey responses between award recipients and unsuccessful award applicants, in all cases these indicated advantages for recipients. Most differences were slight, however. Out of 18 areas for training, a meaningful advantage for recipients is indicated in only four, mostly for SSHRC-funded students at the doctoral level in areas related to dissemination and collaborations, as illustrated in Figure 5.

Description of figure

Figure 5 is a heat map summarizing differences in reported training exposure between students funded through agency direct stream awards and students who had applied unsuccessfully for these awards. Results are reported for 16 areas of training, including, for example, technical research skills, professional skills and experience with multidisciplinary collaborations. Figure 5 shows where there were statistically significant differences in average ratings by recipient and unsuccessful applicant subgroups, organized by the agency and the student’s degree level. Significant differences of 6-10 percentage points between groups are categorized as “slight”. Average ratings that are 11-15 percentage points apart are categorized as “somewhat different”. Differences of more than 15 percentage points are categorized as substantial.

Results:

No statistically significant differences were found among recipient and applicant groups for technical research skills, critical thinking skills, creative thinking skills, professional skills, teaching and mentoring skills, preparing grant proposals, and multidisciplinary collaborations.

Differences were found in other types of training experience as follows.

For SSHRC-funded doctoral students, average ratings between recipients and unsuccessful applicants differed substantially for opportunities to prepare papers for publication and to prepare conference presentations. Average ratings differed somewhat between groups at the doctoral level for participating in conferences and experience with collaborations outside Canada. Slight differences in ratings were found between groups at the doctoral level for non-technical research skills, participating in workshops, experience with multisectoral collaborations, experience with collaborations outside Canada, and mobility during graduate students. Note that this last difference applies only to CSG D recipients, as SSHRC-funded doctoral fellowship recipients did not differ in their rating from unsuccessful applicants.

For SSHRC-funded master's students, there was a slight difference in experience with collaborations outside Canada. Otherwise, no differences were found among recipient and applicant groups at the master's level.

For NSERC-funded doctoral students, average ratings between recipients and unsuccessful applicants differed slightly for preparing papers for publication, preparing conference presentations, and experience with multisectoral collaborations.

For NSERC-funded master’s students, there was a slight difference in non-technical research skills and experience with multisectoral collaborations.

For CIHR-funded doctoral students, average ratings between recipients and unsuccessful applicants differed slightly for preparing papers for publication, preparing conference presentations, participating in conferences, experience with multisectoral collaborations and mobility during graduate studies.

For CIHR-funded master’s students, there was somewhat of a difference in experience with collaborations outside Canada. There was a slight difference in non-academic communications and experience with multisectoral collaborations.

Note that there were no meaningful differences observed in the results between recipients of the CGS D and recipients of agency-specific doctoral awards, except for SSHRC doctoral award recipients in terms of mobility during graduate studies.

The results indicate some areas of strengths in the training system generally. All students contributing to the evaluation, including award recipients, unsuccessful award applicants and students participating in the indirect stream, reported substantial opportunities to develop technical research skills, as well as critical and creative thinking skills.

Results for nontechnical research skills^{Footnote 52} were weaker across all groups. Opportunities to develop teaching and mentorship skills were also weak overall, as were opportunities for training experience outside their institution. Respondents to the student survey who had participated in the indirect stream tended to report greater opportunities for professional skills development than did students in the direct stream.

Overall, only a minority of both direct and indirect participants reported training experiences outside their institution. NSERC’s CREATE participants were a distinct exception. A majority (64%) reported at least one such experience. To the extent that cross-sectoral and international research experience are priorities for the agencies,^{Footnote 53} this indicates areas for agency focus on development.

Within the indirect stream, results on training experiences aligned with the objectives of the three indirect funding types. For example: across the case studies and grant holder interviews and in survey results, stronger emphasis on research skills training and lesser emphasis on professional skills was observed in Type 2 projects. As a reminder, “Type 2” projects are funded through grants for individual researchers or teams of researchers. A relatively stronger emphasis on professional skills was demonstrated across the Type 3 and 4 projects (i.e., grants for research collaborations between multiple organizations, and grants for programs or approaches to enrich student training). Type 2 funding also exhibits strengths in traditional areas of dissemination / communications, including authoring or co-authoring papers for publication and preparing conference presentations.

Type 3 and Type 4 funding may be especially suited to position students for non-academic careers. On the student survey and in case studies, Type 3 and Type 4 FOs showed strengths in non-academic communications, networking activities with potential employers, mobility and collaborations. In case studies, students involved in Type 3 and Type 4 grants emphasized that the opportunities to expand their networks through contacts with industry/government partners were particularly helpful to them, in part because their participation placed them in a more competitive position in the job market. Several students reported that they were recruited for desirable jobs by the same employers immediately after completing their degree. For example, students participating in the NSERC Type 4 case study project reported increased awareness of skills relevant to careers outside academia and the value of their skills to industry and other non-academic stakeholders. Students also reported gaining confidence in their ability to apply for industry positions once they completed their studies, citing the training workshops in networking skills as particularly useful for increasing confidence in dealing with industry.

Within NSERC, Type 3 and Type 4 students responding to the survey were less likely to indicate they plan to go on to further academic studies (including postdoctoral), than Type 2. Additionally, in comparison to Type 2, they were more likely to indicate their experience was useful in preparing them for non-academic careers.

Broadly, the data support the rationale for the different types of funding in the indirect stream as the three types demonstrate potential to increase the range of opportunities available for students with different research and career interests.^{Footnote 54} As the agencies develop priorities for research-intensive career training, for example in support of diverse and evolving career paths and to enable international mobility, the detailed results indicate multiple areas of opportunity.

Other expected outcomes of direct awards: independence and prestige/recognition. Direct funding awards are intended to carry additional benefits for students, such as greater independence and recognition or prestige. Having funding independent of supervisors or institutions is intended to allow students more independence to seek out mentors and training environments that best fit their interests. About one-third of master’s award recipients (31%) and 41% of doctoral recipients reported that their award resulted in a substantial (i.e., large or very large) increase in control or independence over their research. Increased independence due to their award was more often reported by recipients in SSH (42%) compared with recipients in NSE or health (32%). Although fewer than half of award recipients reported increased independence, it is potentially very important for the students in this group.

Notably, in interviews, key informants noted that students who are most likely to win awards typically have engaged and supportive supervisors to begin with. Related to independence, award recipients responding to the survey were as likely to report substantial overlap between their thesis research and their supervisor’s research as were unsuccessful applicants.^{Footnote 55} Further, a student’s degree of independence depends on their field and stage of degree, the availability of experts in their field to supervise them, and a student’s ability to relocate, as noted by key informants in interviews. Relocating was also noted to be unrealistic at the master’s level, as the CGS M can only be held at the institution that offered it. Other reasons for the limited impact of awards on student independence are detailed in the companion reports.^{Footnote 56} Notable among them are the amount of support needed by students from their supervisors to prepare a successful application to the agencies’ awards, as reported by students, institutional staff and supervisors, and that the awards represent insufficient funding for financial independence. The latter is revisited below.

Prestige/recognition was reported to be an important outcome of agency awards by about two-thirds of direct award recipients; more often by CIHR recipients (69%) than NSERC recipients (59%).^{Footnote 57} When asked to volunteer open comments, some recipients described the importance of the award for their own confidence or motivation to pursue research. Some students who identify with smaller population subgroups or who were pursuing less conventional methods or topics noted the award’s prestige was important to open doors or convince others of their ability to produce valuable or rigorous research.

Does talent-related funding influence student career trajectories?

Degree progress and career outcomes. The findings indicate that holding an agency award at a lower-level degree provides some advantage in higher level award competitions. However, the advantage is very small. A substantial proportion of award recipients at all levels are first-time applicants or previously unsuccessful applicants. This is a positive finding from the perspective of access, as an indicator that the portfolio provides multiple entry points for students to access agency direct support.

Opportunities to gain exposure to research training early in an academic path can affect student trajectory. This is pronounced for students who gained exposure to research in their undergraduate studies through NSERC’s Undergraduate Student Research Awards (USRA). More detail is provided in a companion report available on request from evaluation@sshrc-crsh.gc.ca.

The findings from longitudinal analysis of agency data also show that students who engage earlier with agency programs have a small advantage in graduate-level awards competitions. The results indicate an advantage to applying for agency direct awards earlier at both master’s and doctoral level (i.e., before program entry or early in degree). While the effect is small, it suggests that students who start on a research-intensive path early in their academic career have some advantage at all levels.

Agency support at undergraduate level is currently limited outside NSERC. A total of 3,150 USRA awards are available each year from NSERC, and 95 each from CIHR and SSHRC.^{Footnote 58} Alongside direct funds through USRA awards, the indirect stream is also a path through which undergraduate students gain exposure to and experience with research prior to graduate studies. Over $100 million of indirect stream funding went to stipends for undergraduate students involved in agency-funded research projects over fiscal years 2018-21. The findings on early advantage reinforce the importance of ensuring that such opportunities are as accessible as possible: this is revisited in the recommendations.

The agencies are playing a key role in training future research professionals in all sectors. Among past portfolio participants who have completed their studies, about half now work outside the academic sector. Further, only about half of portfolio participants still in their degrees are aspiring to an academic career. While the agencies have a central interest in the academic sector, research intensive careers outside academia are also important for Canada’s research capacity. The distinction between academic and non-academic careers is less sharp than it once was, as individuals increasingly move between and bridge sectors. Moreover, academic research benefits from partners outside academia who help recognize and diagnose real world problems, connect academics with places, people and ideas, and who apply research-based knowledge in Canadian society.

For direct stream funding, very little difference was found between agency award recipients and unsuccessful applicants in terms of degree progression and career trajectory outcomes, with a few exceptions noted below. This does not necessarily mean that the agencies’ scholarships and fellowships are not important to students and do not contribute to their training. Rather, it indicates that the agency awards provide limited incremental advantage to recipients compared to their peers who are part of the same training environment and, as reported below, tend to have similar funding profiles. Agency awards do not appear to impact the length of time taken to complete a graduate degree. There is no evidence of a relationship between agency awards and continuing to higher-level academic study or full-time employment postgraduation. Findings were mixed on other career-related outcomes, such as level of position or type of role, with no consistent patterns evident across cohorts of direct stream participants. There is no indication that agency funding influences likelihood to work in Canada or abroad.

A meaningful advantage for award recipients is indicated a few areas: degree completion at doctoral level (SSHRC and NSERC), academic career aspirations among first-generation students (SSHRC and NSERC), and research-intensity of employment post-degree (SSHRC and CIHR). Similar to findings on training experience (above), outcomes are most pronounced for students in SSH. An overview of results is illustrated in Figure 7.^{Footnote 59}

Description of figure

Figure 6 shows a heat map of reported outcomes for degree progression and early career outcomes, comparing ratings by agency direct award recipients and unsuccessful applicants. Results are reported for 10 areas of outcomes, including, for example, degree completion, academic career aspirations, and plans for careers in Canada. Figure 6 shows where there were statistically significant differences in average ratings by recipient and unsuccessful applicant subgroups, organized by the agency and the student’s degree level. Significant differences of 6-10 percentage points between groups are categorized as “slight”. Average ratings that are 11-15 percentage points apart are categorized as “somewhat different”. Differences of more than 15 percentage points are categorized as “substantial.”

No statistically significant differences were found for time to degree completion, past students’ advancing to further studies, academic career aspirations, current students’ plans for careers in Canada, past students’ plans for careers in Canada, and past students’ career sector outcomes.

For SSHRC-funded doctoral students, average ratings between recipients and applicants differed somewhat for degree completion. There was a slight difference for current students’ plans to continue to further studies, for academic career aspirations in the first-generation student subgroup, and research intensity of employment.

For SSHRC-funded master’s students, there was somewhat of a difference for research intensity of employment.

For NSERC-funded doctoral students, average ratings between recipients and applicants differed somewhat for degree completion.

For NSERC-funded master’s students, there was somewhat of a difference for first-generation students’ academic career aspirations.

For CIHR-funded doctoral students, no differences were found among recipient and applicant groups.

For CIHR-funded master’s students, there was a slight difference for research intensity of employment.

Note that there were no meaningful differences observed in the results between recipients of the CGS D and recipients of agency-specific doctoral awards.

Most indirect stream participants in the evaluation were still enrolled in their degrees at the time of the evaluation: equivalent data on career outcomes is therefore not available for all indicators at this time.^{Footnote 60} Where data were available for indirect stream participants, outcomes were the same or better than for agency award holders. For example, indirect stream participants were as likely or more likely than direct award holders to report plans for further academic study after their current degree. For all three agencies, past indirect stream participants were more likely to be working in the academic sector compared to their direct stream counterparts.

Past indirect stream participants were also as likely as their direct stream counterparts to be working in Canada after their degrees. Among students still enrolled, the vast majority of award holders, unsuccessful applicants and indirect stream participants responded that they intend to work in Canada after completing their degrees (95%, 94% and 93%, respectively). The vast majority of international students reported plans to work in Canada (89%), only slightly less than Canadian citizens (96%).

In terms of career readiness, a majority of indirect stream participants gave high ratings on the survey to their project experience for academic career preparation (80%) and non-academic career preparation (68%). Similarly, a majority of direct award holders (68%) gave high ratings to their award in terms of usefulness to academic career preparation. However, only a minority gave high ratings in terms of the award’s value for careers outside academia (39%). As a reminder, only about half of portfolio participants still in their degrees are aspiring to an academic career.

Sufficiency of financial support. Financial support is important in the indirect stream, as the amount of financial support provided can be important to a student’s ability to participate in research projects and acquire the intended experiential benefits. Financial support is even more important in the direct stream. It is fundamental to the logic behind the direct funding. The awards need to make a sufficient financial contribution to affect a student’s ability to focus on their studies, and thereby affect their progress in their research training.

The evaluation considered the sufficiency of the agencies’ financial support in five ways: award and stipend levels relative to cost of living; reported overall net increase in funding after receipt of an agency award; comparison of overall funding reported by award holders, unsuccessful award applicants and indirect stream participants; stability of direct and indirect funding; and reliance on other (i.e., non-agency) sources of funds.

The direct stream’s objectives are clearly not realistic given current funding amounts, and especially considering constraints on recipients’ ability to supplement their awards. The CGS D stands out in three respects: a majority of recipients report a substantial net increase in overall funding and also greater income stability from this particular award. Compared to other students, about a 12% difference in overall funding is reported by CGS D holders compared to students who have applied unsuccessfully for agency awards or received indirect support.

However, nearly all students participating in the evaluation, including CGS D recipients, have to seek out and sustain income from multiple sources to finance their degrees. This undermines the stated intent of the award to enable students to focus on their studies.

Relative to cost of living. The cost of living in Canada has been steadily increasing. Over the period covered by this study (2014-22), the Consumer Price Index increased by about 23%. Average tuition fees for domestic students were 16% higher in 2022 than in 2014 (Statistics Canada, 2022). Despite increases to costs of living and tuition, the dollar values of agency direct awards have not increased in this time period.^{Footnote 61}

Description of figure

Figure 7 shows a bar graph of the average stipend levels for agency awards relative to cost of living for single-person households in Canada in 2019. The average cost of living was $49,203 for a single-person household. The average stipend for CGS D and CIHR DRA/DFSA is $35,000. The average stipend for NSERC PGS-D is $21,000. The average stipend for CGS M is $17,500. The median indirect stipend for each agency is $18,000 for CIHR, $17,499 for NSERC and $11,349 for SSHRC.

None of the funding types in the portfolio provide full funding to students. Based on Statistics Canada’s (2019) average cost of living estimate for a single person household, the CGS D and CIHR’s DRA/DFSA offset roughly two-thirds of a student’s costs. Other agency awards represent less than half, down to one-third for the CGS M and one-third for an average stipend through agency indirect funding, as shown in Figure 7. The funding received from the agencies is proportionately less for students whose costs are higher than the estimated average. This has a differential impact on some student subgroups, including students with disabilities and those with dependents, as well as those in higher-cost degree programs or geographic areas, among others.

Net increase in funding. The net financial contribution of an agency award differs from student to student. Most of SSHRC’s CGS D recipients (86%) and CGS M recipients (77%) report a substantial^{Footnote 62} overall net increase in their funding after receiving their award, as do 75% of NSERC’s CGS D recipients and 71% of CIHR’s CGS D recipients.

However, a substantial proportion of agency award recipients reported no gain to moderate net gain^{Footnote 63} in their overall funding after receiving their award. This includes more than one-third of the agency-specific doctoral awards (23% of SSHRC’s, 43% of CIHR’s and 40% of NSERC’s), one-fifth of CIHR’s CGS D recipients, and about one-third of NSERC and CIHR CGS M recipients.

A majority of agency award recipients (60%) reported reduced access to at least one other source of funding after receiving their award. Among these students, 80% reported reduced access to institutional funding^{Footnote 64}. This observation is in line with data from interviews and case studies about funding reallocation within institutions from award recipients to other students. In interviews with other funders of graduate students, administrative staff at academic institutions and grant holders, as well as in the student survey, the practice of redirecting funds from students who obtain awards to students who do not hold awards was described as routine in some institutions. Funding floors and funding caps were frequently referenced. Funding caps in institutions, where these exist, affect the total amount of funding a student can receive. Where reallocation is practiced, direct funding effectively catalyzes the redistribution of funds across students in a department or in a research team, with intent to maximize the overall number of students with financial support.

Students who had received agency awards were invited to include open-ended comments about changes to their funding. Respondents explained that receiving an agency award meant that they lost some or all their other funding, including institutional funding and provincial funding (scholarships and/or student loans). Some recipients report being required to reduce external employment hours.^{Footnote 65} Some students expressed a view that reallocation is a fair policy as it provides funding for other students. However, others felt that reallocation was also about the institution saving money or recruiting more students, and that this is not fair. A variety of concerns were raised by student respondents, including that the practice can create the perception that some research units are sustaining an inexpensive labour force by effectively capping overall remuneration.

A known challenge associated with scholarship reallocation is that students in financial need may lose access to support they need to finance their studies (Jaschik, 2017). Some students who received direct awards reported finding themselves in a challenging situation where they were required to complement what the agencies provided, because agency award funding fell short of the cost of living, yet at the same time they faced limitations on their access other forms of support.

Availability of information about reallocation practices is another important concern. Many students responding to the survey for this evaluation described a lack of awareness of policies or regulations about the impact of an award on their access to other sources of funding. Some students reported that the loss of their other funding was unexpected and disruptive. To the extent that policies around funding reallocation are not well known, not sufficiently transparent or inconsistent, students are less able to make informed choices about the time investment to apply for and/or the consequences of accepting an award. A few survey respondents described challenging the loss of funding with their institution, and that as a result, they had their funding restored. Other students described a reluctance to voice concerns to their supervisors or departments for fear of repercussions, either during their studies or later when they seek to establish a career in the same field of research.

Specific practices vary by institution and by faculty/department within institutions. They are also understood by students to vary among individual academic supervisors. The data suggest that they vary by category of student (for example, out-of-province or international student status, by tuition level). In practice, the variability creates challenges for individual students and for stakeholder organizations to understand impacts. It was beyond the scope of this study to measure or document them comprehensively. Further study of funding reallocation practices and the impact on students―particularly on low-income students―is indicated.

Overall funding. On average, total financial support per term reported by agency award holders is about 10% higher than reported by unsuccessful applicants and 20% higher than students funded indirectly by the agencies who have not applied for direct awards.

Stipend levels from agency indirect funding vary widely from student to student. The overall average stipend reported by students is just above what is offered by most provincial scholarships. Notably, no relationship was found between the size of an agency grant and the stipend amount paid to students.^{Footnote 66} Rather, higher value grants appear to result in more students receiving funding. This indicates that increasing the size of agency grants will not, on its own, result in increased stipend amounts. Also, although the agencies require a majority of Type 4 funding to be allocated to trainee salary (NSERC’s CREATE) or training activities (SSHRC’s PRTI), no statistically significant difference was found between Type 3 and Type 4 stipend levels. This suggests that mandating overall proportions will also not necessarily translate to increased stipend levels.^{Footnote 67}

Average stipends are significantly higher in NSE and health compared to SSH. Notably, while most indirectly funded students in NSE (78%) and health (71%) reported substantial overlap between their thesis and the project funded through the research grant, only about half (46%) of indirectly funded students in SSH did so. This suggests that for many SSH students, their agency-funded project participation is additional to or beyond their thesis work. In other words, that indirect stream students in SSH may need to invest additional effort for less funding, compared to NSE and health.

Stability of funding. A majority of award recipients reported increased income stability as a result of receiving their agency award. Echoing the results for net financial contribution, SSHRC award recipients were especially likely to report a substantial increase in income stability (78%). CIHR and NSERC recipients were somewhat less likely to do so (64% and 59% respectively).

Overall, income stability was the highest rated benefit of an agency award, higher than the amount of funding, prestige, independence or supervisor support. In open comments to the survey, students described the stability of the funding as important in its own right for their ability to focus on studies and to plan for the future because of the reduced need to seek or apply for funding year-to-year, a process that requires substantial time investment with uncertain results. This underscores the importance of award duration as a potential contributor to intended outcomes.

Agency awards are shorter than average degree duration. The CGS M is one year in duration whereas the average master’s degree completion time is 2.2 years. Some provincial programs offer longer duration awards, for example, Quebec, Manitoba, Ontario and Nova Scotia offer two-year scholarships to master’s-level students.

The three-year term of a CGS D award is shorter than the average doctoral degree duration by 1.8 -2.6 years. Average doctoral degree durations are 5.6 years for SSHRC participants, 5.0 years for CIHR and 4.8 years for NSERC, with no difference in degree duration between award holders and other students. Some subgroups of students report longer degree durations, which translates to longer periods in which they must secure other funding beyond the term of their award. For example, SSHRC students self-identifying with disabilities report an average doctoral duration of 5.9 years. This is close to twice the duration of the CGS D, as illustrated in Figure 8.

Description of figure

Figure 8 shows a bar graph of average duration of doctoral studies and the proportion of doctoral studies covered by agency award, by domain, scholarship, and student’s self-identified disability. The CGS D award is a 3-year term. Students reported a SSHRC doctoral term lasts 5.6 years on average, 5 years for CIHR, and 4.8 years for NSERC. SSHRC students self-identifying as having a disability reported an average doctoral term of 5.9 years; NSERC students reported 5.1 years. The number of academic terms beyond the award duration is 7.5 for SSHRC, 8.7 for SSHRC for a person identifying as having a disability, 6 for CIHR, 5.5 for NSERC, and 6.3 for NSERC for a person identifying as having a disability.

The agencies set the amount and the duration of direct awards. Amount and duration of indirect funding to individual students is determined by the grant holder and influenced by multiple factors including policies of the institution, duration of the grant and funded research, and level of involvement of the student in the project. The average duration was just over two years. Most indirectly funded students report receiving financial support from the grant holder for only a part of the duration of their involvement in the project.

Reliance on other sources. Almost all students participating in the portfolio (92%-97%) rely on funding from multiple sources. There is little apparent difference in the non-agency types of funding accessed by award recipients compared with unsuccessful applicants and indirect stream participants. One exception is student loans. Agency award recipients were less likely to report accessing student loans, with a difference of about 12 percentage points compared with unsuccessful applicants. However, it is important to note that lesser use of student loans may be due to reduced need, or reduced ability to access loans after receiving an agency scholarship.

Agency award recipients reported off-campus employment in similar proportions to unsuccessful applicants. However, the amount of reported off-campus income is lower on average for recipients with active awards. This suggests they may be working somewhat less off campus (i.e., fewer hours) compared with unsuccessful applicants, while their awards are active. Although award recipients and unsuccessful applicants were equally likely to report receiving non-agency scholarships, agency award holders reported a significantly lower amount of funding from non-agency scholarships compared to other students. Students in SSH were more likely than students in NSE or health to report income from all types of employment, including on campus and off campus. SSH students were also more likely to report use of student loans.

While there is no question that agency direct awards and stipend levels paid through research grants have not kept pace with―and are now far below―students’ cost of living, increases in award values and/or grant sizes alone are not likely to address student funding constraints or achieve agency objectives for training at student level. Raising the value of agency direct awards may increase individual financial benefit but only to the extent that award value is raised high enough to sufficiently offset reallocation of and reduced eligibility for complementary sources of support. There was no evidence of larger research grant sizes leading to higher stipends for the students involved. Finally, both the amount and the stability of funding are important considerations. This is revisited under recommendations.

What should the agencies consider in the context of changes to the training environment and/or student experience due to COVID-19?

The impact of the pandemic was still uncertain at the time that data collection was underway for this evaluation. To better understand the impact on student training, qualitative data were collected from multiple perspectives. Graduate students, academic supervisors, academic staff at the institutional level and other funders of graduate students participated in semi-structured interviews. Students and grant holders responding via survey were also asked to include comments about the impacts of the pandemic and its implications for the near term (i.e., 1-2 years). External data were also reviewed (StatsCan, 2020).^{Footnote 68}

The most commonly identified impacts across data sources related to the following: delays and obstacles in data collection and thesis/degree completion due to the closure of lab and facilities, as well as reduced networking and hands-on training opportunities for students during the pandemic period. Adopting virtual learning resulted in the cancellation of workshops and the modification of internships. The lack of hands-on learning and in-person interactions with peers and senior graduate students were reported by students to have greatly diminished the quality of their training. Additionally, mental health and financial impacts were identified by all respondents and were reported to affect students’ well-being and academic performance. It is important to note that the pandemic experience was not uniform: different students had different experiences. Pandemic impacts on student trajectory in the future are likely to be both multifaced and varied. A small number of students pointed to the importance of being engaged in agency-funded research projects during the pandemic, as these projects helped to keep them engaged and in contact with other students and researchers when other opportunities for collaborative work were suspended.

In response to the pandemic, the agencies implemented a variety of measures to support graduate student training.^{Footnote 69}

Going forward, the agencies may be able to contribute to efforts within institutions to help graduate students and early career researchers address deficits in training and qualifications stemming from differences in opportunities available to them during the pandemic, relative to those who completed their degrees before or after. Over the next few years, the agencies may wish to consider support or incentives to encourage professional networking, knowledge mobilization and publishing for senior graduate students, postdoctoral fellows and early career researchers. The considerations noted above may be particularly relevant to review processes for research grants that are aimed at early career researchers. Overall, the recommendations below to improve access to training support for students on nonlinear or interrupted paths through academia would also benefit students whose studies were most disrupted by the pandemic.

Recommendations

This evaluation was designed to support strategic decisions about the agencies’ funding for research training, and to identify opportunities to strengthen the agencies’ efforts across the portfolio of funding. This purpose is reflected in the recommendations. The evaluation’s findings indicate opportunities for the agencies to diversify their reach and to renew their contribution to research training: this is the focus of the recommendations listed in Table 2 and detailed below. Where recommendations touch on the larger system of student funding, they focus on aspects of the system that impact the students participating in agency programs.

Table 2. Recommendations

Empowering trainees

The new Tri-agency Training Strategy aims to support, inspire and empower the next generation of leaders. Students deserve greater transparency and better information about the funding system. This is a first essential step to empowering trainees, and it will contribute to informed discussion and community engagement on research training going forward. Given the decentralized nature of student funding, the federal research funding agencies are uniquely placed to provide leadership.

1. Improve agency communications and public reporting about agency investments in training for graduate students. Provide a centralized source for accurate and timely, whole-portfolio information.

   Beginning with the agencies’ own funding, the agencies can model their commitment to empowering trainees by improving the clarity and utility of public information about their funding for research training. The agencies’ scholarship and fellowship awards are highly visible and they take centre stage in public discussions about agency funding for trainees. However, some aspects of direct funding are less well understood, including its reach and its basis in a contribution (partial funding) model. The agencies’ indirect funding for training is less well understood outside the agencies. Existing public reporting occurs at agency and program level: information about trainee support is therefore distributed in multiple locations on agency websites. Consolidated reporting on the tri-agencies’ trainee support would provide a single source of accurate and timely information for stakeholders.

2. Work with academic institutions to improve transparency and clarity of policies and practices related to funding reallocation.

   Institutional policies and practices on funding reallocations that affect agency award holders should be explicit, transparent and clearly communicated to students, including levels of any funding caps and the types of funding that are subject to reallocation. Some variation in policies and practices between and within institutions may be necessary given that local conditions and contexts vary. Flexibility to redirect funding can benefit students as outlined above. However, greater transparency and clarity is important to help students make informed decisions about whether to apply for external awards and which to accept, and to support their overall financial planning over the years of their degrees. The agencies and academic institutions have a common interest in preventing unexpected changes to students’ financial situation, which can impact a student’s training experience. To ensure students have the information they need, the agencies should request that institutions provide clear, publicly accessible information (where not already available). The agencies could signal in their own information to prospective awards applicants that they should check policies at their institutions and with other funders about the potential for reduced access to complementary sources of funds should they receive an agency award.

   Greater transparency and clarity will also better position students to self-advocate where needed and allow for more informed debate on this aspect of the graduate student funding system. Based on student input to the evaluation, limitations in available information and/or awareness about reallocation practices negatively affect how graduate students perceive the research funding system and research funders. It can lead some students to not see a future in research. This is important so that all parties (students, institutions and funders) can better appreciate the strengths of the funding system as it exists and contribute to improving on its weaknesses.

Providing leadership to improve funding practices for students.

3. Work with academic institutions to reduce the impact of reallocations on students and address underlying factors driving funding reallocation practices.

   As a longer-term measure, the agencies need to collaborate with institutions, provincial funders and student organizations to mitigate the impact of reallocations on students―particularly those in greater financial need―and to address underlying root causes. Although addressing the root causes of weaknesses in the student funding system is beyond the scope of the agencies alone, it is of shared interest with other funders, academic institutions and students: the agencies can facilitate improvements.

   Funding reallocation presents a challenge for funders whose funding is intended to contribute to student financial well-being and thereby contribute to student outcomes but is not intended to be full funding for the student. It dampens the contribution the agencies make to a student’s financial well-being and is a potential disincentive for external funders to invest in direct awards. The impact is extraordinarily difficult for the agencies to measure or monitor due to wide variation in practice and lack of transparency. Under these conditions, it is difficult for the agencies to effectively identify solutions and design effective interventions for student support.

   • Under no circumstances should agency scholarship recipients experience no net increase or lesser net funding than before receiving their award. The agencies should ensure that guidelines are in place within institutions to protect against this possibility and that institutions provide a process to students to formally challenge reallocations should they occur.
   • The agencies should implement monitoring on reallocation as it affects students funded by the agencies, for example through routine follow-up surveys of award recipients.
   • The agencies can play a leadership role in working with institutions to ensure funding caps are not set too low and / or a minimum “living wage” funding floor is established below which reallocation and reduced eligibility for other funding should not occur.
   • The agencies can work with providers of needs-based funding (national and provincial) to ensure that scholarship students in financial need can continue to access needs-based grants and loans.
   • Further study of the root causes underlying funding reallocation practices is recommended as a step toward identifying solutions. Potential drivers include (but are not necessarily limited to) overall financial constraint in the student funding system, funding system fragmentation and gaps, and student recruitment pressure on institutions. The agencies could consider commissioning such a study.

Develop a whole portfolio strategy.

4. Future development of a strategy for research training should encompass the whole portfolio.

   No single program will be able to do all things. A whole portfolio strategy is needed given the importance of interactions between the direct and indirect streams of funding, and the agencies’ ambition to make progress on complex, system-level challenges like EDI and training for multisectoral careers. The agencies need to leverage the full portfolio, in a synergistic way, to meet these shared priorities.

   The direct and indirect funding do not operate in isolation from each other. Some of the best outcomes are demonstrated for students who access both types of funding, as described in the detailed companion reports. Indirect funding can provide essential research experience for students to later develop competitive applications for scholarships. It can also fill gaps between the funding terms of direct awards.

   In concert, the two streams show potential to expand the range and variety of access points to federal support for students from diverse backgrounds and with diverse research interests. It is particularly critical that the complementarity of the different types of agency funding be retained and leveraged so that each type of funding does not have to try to achieve all aims, and that the funding be leveraged strategically and holistically, as a whole portfolio, to diversify the agencies’ reach.

   Developing a training strategy that considers the full complement of agency funding for trainees would also support better delineation of the agencies’ role and objectives with respect to careers outside academia. As noted above, while the agencies are particularly interested in the academic sector, they are already playing an important role in developing the broader research ecosystem. Research-intensive career is a term used by the agencies in multiple documents. However, at the time of this evaluation, the agencies did not have a common definition of what a research-intensive career encompasses in the context of agency training objectives. A working definition was created for this evaluation that can provide a starting point.^{Footnote 70} A fuller articulation will be important to help the agencies identify the bounds of where their support is best placed and where to prioritize focus.

   This evaluation was the first time the agencies have undertaken a horizontal study of their investment in student training across individual program areas. The depth and richness of findings are a result of a very fruitful collaboration across the three agencies. Through the strategy development process, the agencies have a unique opportunity to continue this collaboration in pursuit of common objectives. Solutions won't necessarily be identical in all research domains, as evidenced by marked differences in SSH, NSE and health in the areas of student access, experience and trajectory. However, as shown in this evaluation, experimentation within one agency can produce useful learning for the others.

   As part of their overall efforts to advance EDI in the research system, the agencies would be able to consider how to optimize reach to underrepresented populations through the whole portfolio (Mayne, 2017).^{Footnote 71} This would help efforts to further develop existing complementarity and synergy of the different forms of funding to benefit EDI. This valuation’s findings can be used to identify points of intervention to enhance access to support for groups of students along funding pathways. Examples include, but are not limited to, scholarship application rates and success rates at each level of review to understand where to focus new reach initiatives and use of indirect support as a gateway to direct funding.

   The portfolio can also be used to improve access over successive degree stages. Specific recommendations are provided below under each of the two streams of funding to reach more students earlier in their academic studies, at smaller universities, part-time students and those not already on a (typical) research-intensive career path. The whole portfolio can be leveraged to help students sustain research intensive studies through to higher level degrees and, overall, envision a viable future in a research-intensive career.

5. Take measures to increase availability and use of data about trainee support across the portfolio, to enhance monitoring, evaluation and public reporting.

   Continuous and effective learning is a necessary skill for funders working in dynamic environments. The agencies have demonstrated their commitment to learning by going beyond routine monitoring and evaluation on multiple occasions. This horizontal study of the agencies' training portfolio is one example. Effective learning also relies on underlying information structures and systems. These are well developed at the agencies and they provided a wealth of data to this evaluation. By using these data, the following potential improvements were identified:

   For the indirect stream: Extend and coordinate data collection across indirect funding opportunities to improve understanding of student access and training experience, as an input to continuous program improvement and periodic evaluation.

   • Data about student training outcomes are already collected for some indirect FOs. These data are collected from grant holders and, in some cases, from student participants in grant-funded research. The amount and type of data collected varies by FO, which limits its utility at a higher, aggregate level (i.e., across FOs). Adoption of a basic set of key metrics across FOs within each agency is suggested, i.e., a common, limited, subset of questions to grant holders for all FOs, to provide more fulsome information to the agencies through monitoring and evaluation at program and portfolio levels.
   • Including a request for contact information of student participants as part of regular required grant holder reporting should be considered. This would enable the agencies to survey students periodically for in-depth information on access, training experience and career trajectory as part of evaluations. Direct survey of students (or inviting direct participation through other methods) is important for student privacy and confidentiality, including but not limited to self-identification data. There are also important privacy considerations related to collecting student contact information indirectly. This change would require careful planning and implementation, in consultation with the agencies’ Access to Information and Privacy (ATIP) offices.
   • Importantly, additional reporting requirements should be designed to limit increased burden on grantees for reporting to the agencies and/or imposing constraints on student participation and the use of grant funds. The indirect stream provides substantial local-level flexibility, both in how grant holders involve students for experiential training in the funded research and in provision of financial support from grant funds. The flexibility may be advantageous to students, for example, for participation to be tailored to students' capacity during the academic term, their development needs and their research interests.
   • Data are already collected for all FOs from grant holders about grant funds allocated to student trainees (Form 300). Minor adjustments to this instrument (for example, to definitions of terms provided on the instrument) should be considered to improve data reliability.

   For the direct stream: Include data for all applicants in routine agency performance monitoring.

   • Variation exists between agencies in the inclusion of data for unsuccessful applicants before the national competition. Reporting on data for applicants across the major stages of review and adjudication within institutions, from the point when the application is submitted by the student, is critical for analysis of application and success rates across process steps.
   • This is also important for more complete baseline data to evaluate new initiatives. When adding designated awards to the portfolio, for example, the agencies have leveraged existing structures and processes that were designed for a different purpose. This means that any existing barriers for equity-deserving groups may be carried forward to these new initiatives, making baseline and monitoring data critically important.

   For public reporting: See Recommendation 1.

Indirect stream funding

6. Clarify training objectives for agency research grants.

   Training objectives are incorporated at FO level for most, but not all, of the funding covered by this evaluation. In some cases, training objectives are only stated at the level of the umbrella program or among umbrella program performance indicators. The agencies should ensure that expectations for training are clearly articulated for prospective grant applicants in FO documentation and in proposal review criteria for applicable programs.

   For any FOs that do not already include it, consider including:

   • A requirement that a training component be included in the proposal.
   • Enhancing and updating current agency guidelines on contributions to research training and mentoring with reference to skills development aligned with agency priorities, as defined through the training strategy development process, and highlighting the importance and value of mentorship as distinct from specific skills development (see below).
   • Requirement that the training component in the proposal identify the proposed student positions and nature of involvement to aid review of proposals.
   • Enhanced performance measurement through financial reporting to improve estimated level of support provided per student.

   Notably, when indirect stream participants were asked on the student survey to describe the most valuable aspect of their project experience for career readiness, mentorship was a cross-cutting theme. About 11% of the total responses across all topic categories focused on the mentoring and coaching support offered by the grant holder as the most valuable aspect of their project participation.

   In the case studies, grant holder interviews and the grant holder survey, some grant holders described pressure from various sources to increase the number of students under their supervision. Among them, some noted a perception that supervising more students increases competitiveness on agency grant applications. To the extent that faculty are under pressure to engage more students on their research teams, this may affect their capacity to mentor individual students. The agencies should review communications to prospective grant applicants and to selection committee members (reviewers of grant proposals) to ensure there is no perceived incentive to increase the number of graduate students to be supported by a grant at the expense of the quality of support for each student. The agencies should consider including explicit guidance for selection committee members to ensure that greater numbers of students are not favoured over quality or depth of involvement, or capacity of grant holders to provide mentorship.

7. Continued investment in the three types of indirect funding is recommended. Consider increasing investment in Type 3 and Type 4 funding opportunities.

   Each of the three types of indirect funding in the portfolio show promise, in different areas.Based on the evaluation findings, continued investment in the three different types is recommended. The three types demonstrate different strengths for areas of skills training and provide a broader range of options important to accessibility^{Footnote 72} and reach to students.

   Type 2 grants account for most of the financial investment in the indirect stream of the portfolio and more than 90% of active grants/year, although proportions vary by agency (see above). Type 3 and Type 4 funding opportunities show promise to help position graduate students for non-academic careers, as well as academic careers that involve collaboration across sectors and internationally. Weaker areas of training that were identified in the findings can be targeted through increased investment in Type 3 and Type 4 indirect opportunities, to the extent that professional skills, mentorship skills, cross-sectoral and multidisciplinary collaborations, and mobility during graduate studies remain important priorities for the agencies. As a reminder, Type 3 and Type 4 funding were not examined for CIHR (see details in Appendix D). Transferability of findings from SSHRC and NSERC programs to health research is uncertain.

   In doing so, it will be important to also sustain existing strengths that are evident in core, in-depth research skills training shown by Type 2 funding.

   Expanding Type 3 and Type 4 funding should not be at the cost of Type 2 funding, as Type 2 funding serves an important purpose in supporting in-depth technical research skills training and supporting students pursuing an academic career path.

8. Consider raising grant levels, alongside measures to encourage an increase in the proportions of grants allocated to training and a corresponding increase in per-student stipend levels.

   The agencies do not control or offer guidance on form, amounts or duration of funding to students through the indirect stream, and students may not be aware that the funding they receive is from the agencies. Decision-making about use of indirect funding is highly decentralized, local and flexible. This is a strength from the perspective of allowing institutions, faculties, departments and supervisors to tailor use of funding to local context. Indirect funding allows for supervisors to hire someone who is the best fit for the position and/or interest in the research and with less reliance on GPA. The flexibility of indirect funds may also allow grant holders to better accommodate personal circumstances of students under their supervision.

   Average stipend levels are far below cost of living, however, meaning that students are required to find supplemental sources of financial support. Stipend funding is also generally less stable than direct funding. Overall, indirectly funded students report less overall funding (from agency and non-agency sources combined) than do unsuccessful applicants to agency awards (with no known agency funding). Participating in agency-funded research projects represents a substantial time commitment for students. Some students in the case studies identified this as a barrier to participation when stipend amounts are too low, pointing to the need to take on other paid work to make ends meet, cover the cost of care for dependents, and relative financial insecurity as factors in their decision to not participate in an agency-funded research project. This can result in students with financial constraints having to choose between research experience opportunities and financial well-being.

   Students in SSH reported lower average stipends than students in NSE and health and were more likely to report income from all types of employment, including on campus and off campus. They were also more likely to report use of student loans. Combined with findings on higher proportion of SSH direct recipients reporting net increase as a result of award (compared to students in NSE and health), the data suggest particular gaps in funding for SSH students.

   Raising grant levels would provide more funds for researchers. However, there was no evidence found in this evaluation of larger grant sizes leading to higher per student stipends. A tendency for available funds to be spread to more students may be due to incentives (or perceived incentives) to increase student recruitment (see above). Stipend levels reported by students also vary widely. The agencies will need to work collaboratively with institutions to identify the most appropriate methods to ensure that increased grant funding, where it is intended to contribute to student financial support, translates to higher average per student stipends, while recognizing the importance of flexibility for grant holders.

Direct stream funding

The direct stream awards included in the evaluation include the Tri-agency Canada Graduate Scholarships (CGS M and CGS D), as well as the agency-specific doctoral fellowships. The agency-specific doctoral fellowships follow a joint application and review process with the CGS and have similar program logic and objectives (for example, foster interest in advanced studies, and related to research skills training and retention of HQP for careers in Canadian academic, public and private sectors). Recommendations apply to all direct FOs except where explicitly stated.

9. Reassess direct funding and clarify its role as part of the future portfolio.

   With exceptions, the intended outcomes of the direct awards as articulated in program documentation are not being met. Given that the direct awards require a separate administration at both institutional and agency levels with associated costs, due deliberation on potential benefits of the awards relative to the cost is warranted. Notably, this is the third evaluation to recommend that the CGS objectives be reviewed due to weak evidence of outcomes (Circum Network, 2008; CIHR, 2016).

   Additional investment to address the current insufficiency of award levels may improve outcomes for graduate students. However, findings from this and other evaluations suggest that the amount of funding provided to award recipients is a necessary but insufficient condition.^{Footnote 73}

   Other changes are needed to the direct awards to increase accessibility and address weaknesses in the existing review process. These are listed below. As part of their deliberation, the agencies will need to take into consideration the financial investment and other resources that would be required to make these changes to the direct awards.

   It is important to point out that the agencies’ scholarships and fellowships date back decades. The research training ecosystem has changed dramatically and new commitments have been made, notably to advancing equity, diversity and inclusion. In sum, it is recommended that the agencies deliberate carefully on what they want to achieve with the awards going forward, what is realistic for this type of intervention to achieve in the future, and whether the necessary investment of time, focus and money to renew the awards is feasible.

   As a first step, the agencies should consider the benefits of the direct awards (summarized below), as well as the role the direct awards can play as part of a larger portfolio (i.e., the distinct contributions of direct awards relative to indirect funding).

   As part of the larger portfolio. Because the student population is very diverse, one solution will not serve all students. In some areas, the evaluation has found the strongest outcomes are indicated for the small minority of students who have participated in both streams of support, either in tandem or sequentially. In short, it is essential that the function and objectives of the direct awards be considered as part of a portfolio of funding. This is precisely the value of a strategic initiative, as it positions the agencies to work with all tools at their disposal.

   EDI. The agencies’ awards show potential to advance the agencies’ EDI commitments in specific areas relative to indirect stream funding.

   Relationships inside the institution are an important enabler for students to apply for direct funding, and the absence of strong relationships can be a barrier. However, existing relationships are even more important for access to indirect funding, as indirect funding is almost entirely mediated by grant holders funded by the agencies. This means that students whose supervisors are less likely to receive federal agency funding are themselves less likely to access indirect support. The direct awards are an alternative pathway for students. In addition, access to indirect support is largely through informal channels, and participation in the indirect stream may be more difficult for some students where low stipend amounts and stability of funding present barriers, combined with the time commitment required to participate in research projects that do not overlap with their thesis. In short, access to indirect funding is influenced by structures in a different way than the direct funding. Having two streams likely increases diversity in agency reach. Lastly, the agencies have relatively more influence over, and ability to monitor, the direct awards processes.

   Of note:

   • Representation of some student subgroups is higher in the direct stream than in the indirect stream, including women, students identifying with disabilities, francophone students, among others. This relates to differences in the characteristics of direct awards compared with indirect funding.
   • Receipt of direct support may be more important for first-generation students to inspire them to pursue academic careers, as reported in the section on career trajectory above.

   Other benefits of direct awards. The student training and funding system has changed since the agencies’ awards were designed; the logic and objectives of the agencies’ awards are now several years old. The existing documentation on intended outcomes of the agencies’ direct awards may understate their potential. For example:

   • Increased income stability is an important benefit of agency direct funding, in addition to the dollar value of the award. Control over award durations is a unique feature of direct funding compared to indirect funding.
   • The agencies’ greater control over the direct stream funding may offer potential to address other priorities for student training, such as international experience, interdisciplinary training and training in emerging or priority research areas. Unlike indirect funding, this level of control also offers potential to direct funding at specific degree levels or to designated equity-deserving subgroups.
   • Outcomes for award holders are indicated in a few areas of training experience and career outcomes as noted above, including research-intensity of employment post-degree. Otherwise, although award holders reported opportunities to develop their skills to a similar extent as unsuccessful applicants, this does not necessarily mean that awards do not contribute to recipients’ training. It does indicate that the agency award holders have similar training experiences as their peers with similar financial profiles.
   • In qualitative comments, some award holders on an academic path made clear links between the financial support from their award and less tangible benefits, such as reduced stress and increased well-being, capacity to produce higher quality work, and increased opportunities to participate in other activities related to their career path (e.g., volunteering).
   • Synergies with indirect funding could also be explored. For example, SSHRC doctoral-level recipients with both direct and indirect support were more likely to intend to pursue further studies than their counterparts participating in either stream alone (41%)^{Footnote 74}.

   In reassessing the direct awards, the agencies need to also consider the following recommendations and the availability of resources to implement them. These recommendations are important if the awards are to align with agency commitments to EDI and if they are to make the desired contribution to research training outcomes.

10. Consider expanding the number of awards at earlier degree levels.

    The agencies could diversify their reach by increasing the number of CGS M awards, as well as research opportunities for undergraduate students, while ensuring that these opportunities are as accessible as possible.

    Students in earlier stages of their academic studies (e.g., undergraduate and master’s) have fewer opportunities to access agency support compared to students in more advanced stages of their academic path (e.g., doctoral students). In other words, the agencies’ funding is oriented to those who are further along in the “pipeline.” Limited reach at earlier degree stages makes the agencies’ awards less able to attract students into research-intensive studies who might not otherwise make that commitment.

    Overall, master’s students report more use of student loans and less access to non-agency scholarship funding than doctoral students. This is important because, as noted in key informant interviews with student supervisors, reliance on loans in master’s degree can affect a student’s ability to continue studies at doctoral level.

    Findings on advantage in CGS M funding for NSERC’s USRA recipients suggest exposure to research during undergraduate studies can be important to prepare students to apply earlier and prepare stronger applications for master’s level awards. Students have a limited time window to apply for CGS M, either in the last year of their undergraduate studies or early within the first year of their master’s-level program (for second year funding). Aside from differences in awareness of the agencies’ awards among undergraduate students, students typically rely heavily on support networks inside academia to prepare their applications. As such, the limited application window is problematic for students who are less likely to have established support networks at an early stage. Given findings on early advantage to award applicants, access points for agency direct support favours students with access to information and support networks before and early in their academic studies, which has implications for EDI.

    Undergraduate funding was not studied under this evaluation. However, the findings on student trajectory indicate that NSERC’s USRA students are ready to engage with agency graduate awards competitions earlier and have an advantage in the CGS M. The USRA acts as an entry point to gain research experience for students in undergraduate studies who may be interested in a research-intensive graduate program. It could be a valuable tool to diversify participation in the graduate level portfolio.

    However, if expanded, it would be essential to ensure that undergraduate level awards do not further widen differences in advantage among students entering graduate studies. Therefore, if the agencies expand the USRA, it needs to be maximally accessible, for example by continuing to offer designated USRA awards for underrepresented groups, students in small universities, and students without first-class GPA standing. In addition to increasing the number of awards, the value of the USRA needs to be considered. A few grant holders participating in this evaluation raised concerns that some students may need to forgo a USRA to seek out higher paying employment to meet their financial needs.

11. Extend doctoral degree stage eligibility windows or eliminate degree stage eligibility restrictions.

    Short eligibility time periods for direct funding, for example, make it more challenging for students who lack research experience in earlier degrees and who need time to catch up with peers. This includes students who did not access research experience opportunities during undergraduate degrees or who were previously in professional graduate programs. It can also make direct funding more difficult to access for those who have had interrupted or nonlinear academic paths and are less ready to apply for agency funding before or soon after their program starts. This compounds the impact of an emphasis on academic history in awards selection criteria, including prior publications and GPA.

    Increasing access for students on nonlinear or interrupted academic paths warrants a priority focus in the intermediate term given the pandemic’s impact on training.

    CGS D applicants typically must apply within 24 months of starting their program. Students on nonlinear or interrupted academic paths are less likely to have acquired experiences necessary for a competitive application [Ref: Trajectory].

    In contrast, those applying to SSHRC’s DF award can apply up to 48 months of full-time study into their program. This provides more opportunity to students who need time to build a competitive academic CV, for example, students who have had fewer opportunities at earlier degree levels for academic publication or to gain research experience relative to peers [Ref: Trajectory]. As was recently highlighted in a policy memo, this also helps reduce emphasis on a student’s accomplishments before their doctoral studies (Baskaran, et al., 2021).^{Footnote 75} Emphasis on prior academic history in awards selection has been raised as a barrier for students on nonlinear paths and students from underrepresented groups.

    An incentive would remain for students to apply earlier in their degree programs to receive more years of funding, particularly if the duration of the CGS D were extended (see below).

    Less than half of CGS (doctoral and master's) applicants agreed that eligibility requirements were easy to understand (Ref: Access). Eliminating degree stage eligibility restrictions would greatly simplify these requirements and would also reduce administrative burden.

12. Allow for part-time students to apply for agency awards without the need to justify their part-time status. This includes eliminating the need for exceptional approvals. Review program documentation for implicit bias.^{Footnote 76}

    The agencies’ overall reach to part-time students is very limited, at about 3% of all students in the portfolio. For context, estimates of part-time enrollment in the overall graduate student population range from 13% (CGPSS, 2019) to 23% (PSIS, 2019, 2021).

    Part-time students can access agency direct funding on an exceptional basis, which requires approval. While some information on part-time status is available, framing as an exception requiring approval and the need to complete additional tasks may send the message to part-time students that they should not or even cannot apply for awards.

    Because direct awards generally require the recipients to be full-time students, students who may need to enrol part-time are less able to access these awards, whatever their reasons for not pursuing full-time studies. These reasons may include circumstances such as financial difficulties leading to the need to work or having dependents requiring care. Part-time enrollment is higher for some groups, including students with dependents, students identifying with disabilities and women (CGPSS, 2019; PSIS, 2021).

    Part-time students are also rare among indirect stream participants. Some indirect opportunities require substantial time commitments. As such, indirect opportunities are not necessarily a viable alternative to access agency funding for part-time students.

13. Address insufficiency of awards funding.

    The direct awards need to provide enough funding to enable recipients to focus on their studies. To the extent that direct awards are also intended to contribute to student independence and mobility, award values need to adequately contribute to financial independence. Ensuring sufficiency of funding includes considering both the funding amount and the duration of funding. At minimum, the agencies should consider:

    • Increasing the value of the SSHRC Doctoral Fellowship and NSERC Postgraduate Scholarship to match the CIHR DRA and CGS D
    • Increasing the duration of doctoral awards to four years
    • Increasing the duration of the CGS M to two years

    With respect to the amount of the CGS M. The evaluation findings do not point clearly to a specific amount. The poverty line for single individual households in urban areas larger than500,000 people is $22,060 (StatsCan, 2022). The recommended increase to the SSHRC and NSERC-specific doctoral awards represents a 40% increase. A commensurate 40% increase in CGS M would bring it to $25,000. As a third reference point, recent external surveys of Canadian graduate students have reported recommended award value for CGS M at $21,000 (SPE, 2019), and an “ideal” value of $28,600 for an average master’s student stipend (OSPN, 2022).

    The reallocation of funds away from award recipients has a dampening effect on the financial contribution of agency funding for individual students (within the direct stream). Together with recommendations related to reallocation of funding (above), the agencies could consider requiring a contribution of funding from institutions for award recipients to make up for some of the remaining difference between award values and cost of living.

    Increasing the length of the CGS M award to cover the two-year anticipated duration of a master’s degree and of the CGS D to cover at least four years of a doctoral degree would also alleviate funding gap challenges and support stability. Members of some underrepresented groups may be especially likely to benefit from the longer award duration.

    Although increases in award values would be recommended unless there are sufficient funds to fully offset the cost of living for recipients and to increase award values with inflation into the future, the agencies’ awards will always be partial funding. Not recognizing this explicitly in the program theory invites an overestimate of the capacity of the awards to impact individual student training experience and trajectory.

    It is important to note that there is a paradox here. Raising award value without increasing the number of awards is likely to intensify their competitiveness. An increase in applications resulting in increased administrative burden, is likely to lead to more reliance on simple metrics like GPA, limit capacity to provide feedback to students, and undermine EDI. This has also been raised by Baskaran et al (2021).

14. Work with institutions to address transparency and perceived fairness of institutional level review processes and to protect privacy of individual students with respect to self-identification / special circumstances. This should include elimination of departmental-level review of applications or, at minimum, departmental-level review of sensitive information on applications.

    The transparency and perceived fairness of the direct funding selection process is a key area of concern for students. It is important to recognize that when describing their experience in accessing agency awards in interviews and on the student survey, students often did not distinguish between institutional processes and agency processes. These distinctions are not always visible to students or may not be relevant from their perspective as these are, ultimately, the agencies’ awards. This means that students’ experience reflects on the agencies and influences their perception of the research funding system, regardless of the level in the process where an issue may occur.

    Some students expressed concern that individual faculty might use their position on selection committees to influence results because they may be able to recoup research grant funds being used as part of a student’s funding package after a student receives external funding, as part of funding reallocation. It is important to clarify that the evaluation does not have evidence that this is the case, rather this recommendation is focused on measures to ensure protections are in place against the possibility, that these are well communicated to students and, overall, for increased transparency. For some applicants, the combination of lack of clarity about the process and limited feedback about results on unsuccessful applications adds to a sense that the process is “mysterious” or biased, which can undermine trust. A few respondents described conflict within departments or among their peers as a result of students’ perceptions of bias in award decisions. The lack of transparency impacts students’ ability to navigate. It can add to confusion (misinformation) and reliance on informal information systems that are not uniformly accessible.

    Concerns about departmental-level selection have been raised by students also with respect to self-identification as a member of an equity-deserving group and documentation of special circumstances in their applications, which can contain sensitive personal information, as noted above. Concerns about data privacy and potential for discrimination due to an agency funding opportunity need to be taken extremely seriously. Eliminating departmental review is one step the agencies can take. The agencies should also consider exploring options with institutions to ensure that institution-level review incorporates privacy protections. An alternative is to consider a national-level competition only.^{Footnote 77}

    Other actions

    As the agencies consider the full body of evidence provided through this evaluation, it can inform other actions, depending on the priorities put forward under the new tri-agency training strategy. If, for example, the strategy prioritizes greater student independence as a result of their award, then the agencies are urged to consider factors influencing student independence as documented, such as non-portability of the CGS M should the student wish to change academic institution.^{Footnote 78}

    As an outcome of the agencies’ reassessment of the awards, they may determine that the changes listed above are not feasible given available resources. If this is the case, the investment should be redirected to alternative modes of support for these students, such as through the agencies’ indirect stream funding, with measures to ensure the funds are allocated to training and with continued monitoring and evaluation (see Recommendations 5 and 8). Above all, if renewal of the awards is not feasible, then it is essential that the current investment for students in research-intensive studies be retained. The agencies’ overarching objective is to foster a rich and diverse research ecosystem in Canada. This will not be possible without the contributions of graduate students, who are current and future members of that ecosystem. For students, it requires an ability to sustain a passion for research through years of intensive study and into a research-intensive career.

    The federal granting agencies have recognized this through decades of investment in research-intensive training, spanning undergraduate to postdoctoral studies. They are now taking stock and preparing to renew that investment, building on what they have learned, to better serve students and the research training system. This is not an easy task in a multifaced, rapidly changing system. The findings from this study―and all the above recommendations―are resources for the agencies to use in that endeavour.

References

Baskaran et al. (2021). Improving the Accessibility of Federal Graduate Research Awards in Canada, Journal of Science Policy and Governance, 18(4). https://doi.org/10.38126/JSPG180405

Bauer, G. R., Churchill, S. M., Mahendran, M., Walwyn, C., Lizotte, D., & Villa-Rueda, A. A. (2021). Intersectionality in quantitative research: A systematic review of its emergence and applications of theory and methods. SSM-population Health, 14, 100798.

Canadian Graduate and Professional Student Survey (CGPSS). (2019). National Report. https://secureservercdn.net/45.40.150.136/bba.0c2.myftpupload.com/wp-content/uploads/2020/08/NATIONAL_CGPSS_2019_ALL.pdf

CIHR. (2016). Evaluation of the Canada Graduate Scholarships (CGS) Program, 2008-2013. https://cihr-irsc.gc.ca/e/documents/evaluation_cgs_program_2016-en.pdf

Circum Network. (2008). Canada Graduate Scholarships (CGS) Program and Related Programs Review.

CRCC. (2022). https://www.canada.ca/en/research-coordinating-committee/priorities/research-talent-for-a-knowledge-based-society.html

Else-Quest, N. M., & Hyde, J. S. (2016). Intersectionality in quantitative psychological research: II. Methods and techniques. Psychology of Women Quarterly, 40(3), 319-336.

Grimes, S., Southgate, E., Scevak, J., & Buchanan, R. (2020). University Student Experiences of Disability and the Influence of Stigma on Institutional Non-Disclosure and Learning. Journal of Postsecondary Education and Disability, 33(1), 23-37.

ISED. (2023). Report of the Advisory Panel on the Federal Research Support System. https://ised-isde.canada.ca/site/panel-federal-research-support/en/report-advisory-panel-federal-research-support-system

Jaschik, S. (2017, July 10). In ‘Scholarship Displacement’ Debate, Who Speaks for Low-Income Students? Inside Higher Ed. https://www.insidehighered.com/admissions/article/2017/07/10/new-law-prompts-debate-over-scholarship-displacement-raising-questions

Mayne, J. (2017). Using theories of reach to enhance equity considerations in evaluation. In Speaking Justice to Power (pp. 133-152). Routledge.

McCall, L. (2005). The complexity of intersectionality. Signs: Journal of Women in Culture and Society, 30(3), 1771-1800.

Mintzberg, H. (1996). Learning 1, planning 0. California Management Review, 38(4), 92.

Moriña, A. (2022). When what is unseen does not exist: disclosure, barriers and supports for students with invisible disabilities in higher education. Disability & Society, 1-19.

Ottawa Science Policy Network (OSPN). (2022). Financial challenges faced by graduate students in Canada: Results from the National Graduate Student Finance Survey. fbeadc_c9aacd9267bf476f90b8a2d44a0e6536.pdf (ottawasciencepolicynetwork.ca)

OCHRO. (2022). Internal report. Office of the Chief Human Resources Officer. CIHR.

Science and Policy Exchange (SPE). (2019). Rethinking federal research funding: Towards more equitable funding for Canada’s next generation. Rethinking Federal Research Funding | sp-exchange

Shpigelman, C. N., Mor, S., Sachs, D., & Schreuer, N. (2022). Supporting the development of students with disabilities in higher education: access, stigma, identity, and power. Studies in Higher Education, 47(9), 1776-1791.

Statistics Canada. (2020a, October 8). Financial information of universities for the 2018/2019 school year and projected impact of COVID–19 for 2020/2021. The Daily. https://www150.statcan.gc.ca/n1/daily-quotidien/201008/dq201008b-eng.htm

Statistics Canada. (2020b, November 25). International students accounted for all of the growth in postsecondary enrolments in 2018/2019. The Daily https://www150.statcan.gc.ca/n1/daily-quotidien/201125/dq201125e-eng.htm

Statistics Canada. (2021). Household spending by household type. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110022401

Statistics Canada. (2022). Low income cut-offs (LICOs) before and after tax by community size and family size, in current dollars. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110024101

The Postsecondary Student Information System (PSIS). (2021, November 24). Statistics Canada. https://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=5017

Treasury Board Secretariat. (2016). Policy on results. https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=31300

WAGE. (2022). Gender-based analysis plus. Gender-based Analysis Plus (GBA Plus) - Women and Gender Equality Canada

Appendix A. List of Acronyms

Appendix B. Glossary of key terms

Active involvement: For the purpose of this evaluation, active involvement was defined as follows: Participating in the planning and/or implementation of the project in a way that might foster 'learning by doing' (i.e., experiential learning). For example, performing specific short-term tasks (e.g., a literature review, data collection activities) or acting in a role with responsibility for many types of tasks over an extended period (e.g., project coordinator).

Co-applicant: An individual, participating in a grant application, who makes a significant contribution to the intellectual direction of the research or research-related activity, who plays a significant role in the conduct of the research or research-related activity, and who may also have some responsibility for financial aspects of the research.

Collaborator: An individual, participating in a grant application, who may make a significant contribution to the intellectual direction of the research or research-related activity, and who may play a significant role in the conduct of the research or research-related activity.

Direct stream: Direct support to master’s and doctoral students is aimed at providing financial support, thereby increasing students’ ability to focus on their chosen area of study by reducing their need to take paid employment.

EDI: Equity is defined as the removal of systemic barriers and biases enabling all individuals to have equal opportunity to access and benefit from the program. Diversity is defined as differences in race, colour, place of origin, religion, immigrant and newcomer status, ethnic origin, ability, sex, sexual orientation, gender identity, gender expression and age. Inclusion is defined as the practice of ensuring that all individuals are valued and respected for their contributions and are equally supported.

First-generation student: A student whose parents or caregivers did not complete postsecondary education.

Full-time study (related term: full-time student): Established according to institutional regulations; full-time hours vary by province and country.

Funding groups

• Direct awards―unsuccessful applicants. Applicants for direct support who did not receive an agency scholarship/fellowship at any point between competition years 2014 and 2021 and were not identified as recipients of agency indirect support.
• Direct awards―recipients. Recipients of at least one scholarship/fellowship from the agencies between competition years 2014 and 2021, who were not identified as recipients of agency indirect support.
• Indirectly funded and award recipients. Recipients of at least one scholarship/fellowship from the agencies between competition years 2014 and 2021, who were also identified as having received indirect support from the agencies.
• Indirectly funded and unsuccessful applicants. Individuals who received indirect support from the agencies between 2020 and 2021 who also applied for but did not receive an agency award between 2014-21.
• Indirectly funded only (non-applicants). Students who received indirect support between 2020 and 2021 and did not apply for agency direct awards.

Funding opportunity (FO): A funding mechanism with specific, self-contained objectives. For CIHR, an FO refers to a competition launch (a call for funding).

Gender: The behavioural, cultural and psychological traits associated with an array of gender identities, including female or male, in a given society.

Grant holder: Nominated primary investigators (NPIs), primary investigators (PIs) or co-investigators for agency research and training grants.

Highly qualified personnel (HQP): Includes undergraduates, graduates, research technicians, research associates, and other technical or research personnel.

Indigenous person: A person who belongs to one of the three Indigenous Peoples in Canada, namely, First Nations, Inuit or Métis.

Indirect stream: Indirect support provides funding to students via grants to researchers and includes FOs for which a portion of the grant is expected to be directed to the training of graduate students.

International student: A temporary resident who is legally authorized to study in Canada on a temporary basis. With a few exceptions, foreign students must get a study permit if they are taking a course of studies that will last for more than six months.

Mother language: Language(s) first learned at home which an individual still understands.

Non-research-intensive program: University programs without a significant autonomous research component.

Part-time study: Constitutes any amount of time less than full-time hours established according to institutional regulations; this amount is based on the accommodation needs of the award holder.

Person with a disability: A person with a physical, mental, intellectual, cognitive, sensory, learning or communication impairment, or a functional limitation, whether apparent or not, and permanent, temporary or episodic in nature, that hinders their full and equal participation in society when they face a barrier.

Primary official language: Official language (French or English) that an individual uses in their day-to-day life.

Principal investigator (PI): PIs have primary responsibility for the overall intellectual direction of research, research-related activities or partnerships. PIs are also accountable for coordinating the grant’s overall financial and administrative aspects.

Research-intensive program: University program with a significant autonomous research component. A significant research component is considered to be original, autonomous research that leads to the completion of a thesis, major research project, dissertation, scholarly publication, performance, recital and/or exhibit that is merit/expert-reviewed at the institutional level as a requirement for completion of the program.

Tri-agency: Tri-agency is an adjective used for activities, programs, policies, etc. collectively involving all three of the federal research funding agencies (SSHRC, NSERC and CIHR).

U15: A collective of 15 Canadian research-intensive universities.

Visible minority^{Footnote 79}: A group of people who have identifiable characteristics that differ from those of the majority or dominant population.

Appendix C. Portfolio funding opportunities

The tri-agencies’ talent portfolio incorporates the primary channels through which the agencies provide support for graduate student training as of 2020/21.

For NSERC and SSHRC, the evaluation included current FOs. Funding that had been awarded as a single one-time grant or which has ended^{Footnote 80} was excluded. A similar approach was taken by CIHR, although CIHR’s scope included some relevant funding through their Research in Priority Areas (RPA) and Investigator Initiated Research Programs (IIR) that are no longer offered as of 2020-21.

As shown, due to differences in CIHR’s funding structure, indirect funding to graduate students from CIHR spans more than one type; individual grants within RPA for example, may fall within Type 2, 3 or 4, depending on the grant. CIHR’s indirect funding was included with Type 2 funding for data analysis in this evaluation.

In the case of the RPA Program, FOs are typically temporary and shorter term in nature because they respond to priorities at a given time. In other words, short term FOs are characteristic of RPA. Therefore, most RPA grants awarded during the period covered by the evaluation (2014-21) were included.^{Footnote 81} The IIR Program’s Foundation Grants were also included, even though these grants sunset in 2019. This was due to the size and importance of this funding, its explicit objective related to training and capacity building, and because funding with similar capacity building objectives was not yet in place.^{Footnote 82}

Funding Opportunity Descriptions

Tri-Agency Funding Opportunities

Canada Graduate Scholarships-Vanier (Vanier CGS): Vanier CGS seeks to support the attraction and retention of top doctoral candidates who demonstrate leadership skills, research potential and a high level of academic achievement in graduate studies in the social sciences and humanities, natural sciences, and/or engineering and health. The Vanier CGS’s objectives include to enable the establishment of Canada as a global centre of excellence in research and higher learning.

Canadian institutions with a Vanier CGS quota are responsible for nominating eligible doctoral candidates and recipients are selected through reviews conducted by the nominating institution and by agency-specific selection committees. The funding value is $50,000 per year for three years. For more information, see: Vanier Canada Graduate Scholarships

The Vanier CGS is not included in this evaluation.

Canada Graduate Scholarships–Master’s (CGS M): The CGS M provides direct funding to students at or entering the master’s level who demonstrate a high standard of achievement and conduct research in the social sciences and humanities, natural sciences, and/or engineering and health. The scholarships are intended to enable the expansion of their research skills and training and contribute to the development of highly qualified personnel.

Qualified students at the master’s level and students in some doctoral programs apply directly to eligible Canadian institutions for the agency scholarship(s) that best align with their field of research. The funding value is $17,500 for 12 months and is non-renewable. For more information, see: NSERC – Canada Graduate Scholarships―Master’s program (nserc-crsng.gc.ca)

Canada Graduate Scholarships–Doctoral (CGS D): The CGS D is intended to support Canadian research excellence and the high-quality research training experiences of doctoral students in the social sciences and humanities, natural sciences, and/or engineering and health by direct funding.

Qualified doctoral students apply through an eligible Canadian institution with an awards quota (or directly to the agency if not yet registered). Students apply to the agency that aligns with their field of research. The funding value is $35,000 per year for 36 months. For more information, see: Canada Graduate Scholarships―Doctoral program (nserc-crsng.gc.ca)

Canada Graduate Scholarship–Michael Smith Foreign Study Supplements (CGS-MSFSS): The CGS MSFSS are complementary to the Vanier, CGS M and CGS D awards. CGS MSFSS provides funding to holders of an active CGS to offset the cost of undertaking research studies outside Canada for a defined period. The funding value is for up to $6,000 for the duration of the period abroad (up to six months). For more information, see: NSERC – Canada Graduate Scholarships―Michael Smith Foreign Study Supplements (nserc-crsng.gc.ca)

SSHRC Funding Opportunities

SSHRC Doctoral Fellowships: SSHRC Doctoral Fellowships are meant to enable doctoral candidates in the social sciences and humanities to fully focus on their studies, seek out the best mentors and contribute to the Canadian research ecosystem.

Qualified doctoral students apply through an eligible Canadian institution with an awards quota (or to the agency, depending on their registration status) and students can hold them at a non-Canadian institution. Students apply to the agency that aligns with their field of research. Note that applicants eligible for both the SSHRC Doctoral Fellowships and the tri-agency CGS D will be automatically considered for both. The funding value is $20,000 per year and is available for 12, 24, 36, or 48 months. For more information, see: SSHRC Doctoral Fellowships (sshrc-crsh.gc.ca)

Insight Grants: Insight Grants support the long-term research initiatives of emerging or established researchers. The grants are intended to enable research that examines and furthers knowledge of complex societal and individual-level issues within and across various disciplines, thematic areas, approaches or subject areas.

Individuals and teams of researchers affiliated with an eligible Canadian postsecondary institution can apply for an Insight Grant. Postdoctoral researchers and PhD candidates can apply under certain conditions. PhD candidates must complete their studies before receiving the grant. Both groups must establish a relationship with an eligible Canadian institution within five months of the grant start date. The funding duration is two to five years, with a maximum allowance of $400,000 over the total funding period. For more information, see: Insight Grants (sshrc-crsh.gc.ca)

Insight Development Grants: Insight Development Grants are meant to enable the early stage research initiatives of emerging and established scholars. The grants support the development of new research questions and methodological, theoretical, and/or conceptual experimentation in eligible subject matter areas.

Individuals affiliated with an eligible Canadian postsecondary institution can apply for an Insight Development grant. Postdoctoral researchers and PhD candidates can apply under certain conditions. PhD candidates must complete their studies before receiving the grant. Both groups must establish a relationship with an eligible Canadian institution within three months of the grant start date. A maximum of $75,000 is available for one to two years. For more information, see: Insight Development Grants (sshrc-crsh.gc.ca)

Partnership Grants: Partnership Grants for large teams of postsecondary institutions and/or organizations of various types that work in formal collaboration. These grants aim to enable the advancement of research, research training and/or knowledge mobilization in eligible areas of the social sciences and humanities by supporting new formal partnerships in their initial stages or existing formal partnerships whereby research activities are new or distinct from previous partnership activities. The quality of training, mentoring and employability plans for students and emerging scholars are an important part of proposed initiatives.

Eligible Canadian universities can apply, and applications are to be prepared by the project director, who must be affiliated with the host institution. Postdoctoral researchers and PhD candidates can apply under certain conditions. PhD candidates must complete their studies before receiving the grant. Both groups must establish a relationship with an eligible Canadian institution within five months of the grant start date. There are two stages to the application process. Applicants that move on to and are successful at the second stage can receive four to seven years of funding, with a maximum allowance of $2.5 million over the four- to seven-year period. For more information, see: SSHRC - Partnership Grants (sshrc-crsh.gc.ca)

Partnership Development Grants: Partnership Development Grants aim to develop research and/or related activities in eligible areas of the social sciences and the humanities by supporting new formal partnerships between postsecondary institutions and/or organizations of other types. Partnership Development Grants enable the design and testing of new partnerships that can result in best practices or models with the potential to be scaled up. The quality of training, mentoring and employability plans for students and emerging scholars are an important part of proposed initiatives. Students and emerging scholars are meant to meaningfully participate in the initiative.

Researchers affiliated with an eligible Canadian institution can apply. Applications may be submitted by a team of researchers and prepared by the applicant/co-director on behalf of the partner organizations. Postdoctoral researchers and PhD candidates can apply under certain conditions. PhD candidates must complete their studies before receiving the grant. Both groups must establish a relationship with an eligible Canadian institution within five months of the grant start date. The funding value is $75,000 to $200,000 over one to three years. For more information, see: Partnership Development Grants (sshrc-crsh.gc.ca)

Partnered Research Training Initiatives (PRTI): PRTIs are intended to enable innovative approaches to the research training experiences of students at the diploma, undergraduate, and graduate levels, as well as for postdoctoral researchers. PTRIs facilitate existing or new initiatives to support the development of research and professional skills among students and postdoctoral researchers, foster climates conducive to networking, and develop collaborative and sustainable research training initiatives that help address research challenges. Partnered research training initiatives must foster the acquisition and development of important skills that complement students’ qualifications and subject matter expertise, and improve their readiness for careers in industry, government, non-governmental organizations and/or academia.

Eligible institutions can apply for the funding opportunity and must submit proposals in collaboration with partner organizations. There are two stages to the application process. Applicants that move on to and are successful at the second stage can receive a maximum allowance of $2.5 million over the four- to seven-year grant period. The majority of funds (at least 75%) must go to support training. For more information, see: Guidelines for Partnered Research Training Initiatives (sshrc-crsh.gc.ca)

NSERC Funding Opportunities

NSERC Postgraduate Fellowships: The NSERC PGS D is intended to enable doctoral students in the natural sciences and engineering to fully focus on their studies, seek out mentors and contribute to the Canadian research ecosystem.

Qualified doctoral students apply through an eligible Canadian institution with an awards quota (or directly to the agency if not yet registered) and students can hold them at a non-Canadian institution. Students apply to the agency that aligns with their field of research. Note that applicants to the PGS D are automatically considered for the tri-agency CGS D, if eligible and vice versa. The value of the PGS D is $21,000 per year for three years. For more information, see: NSERC Postgraduate Scholarships - Doctoral program (nserc-crsng.gc.ca)

Discovery Grants: Discovery Grants are intended to enable the longer-term research initiatives of early career and established researchers by providing support for operating funds. The grants aim to facilitate creativity, innovation and flexibility in the natural sciences and engineering by allowing researchers to pursue research avenues as they emerge and offer the opportunity to address high-risk-high reward topics.

Discovery Grants provide funding for up to five years, and there is no cap on the amount that early career and established researchers can request. For more information, see: NSERC – Discovery Grants program (nserc-crsng.gc.ca)

Discovery Development Grants: Discovery Development Grants are complementary to the Discovery Grants program. The Discovery Development Grants fund researchers from small universities and aim to enable the promotion and maintenance of a diversified base of high-quality research in small Canadian universities and contribute to stimulating research training environments.

Applicants must be from small Canadian universities to be eligible for the Discovery Development Grant. Researchers must have applied to the Discovery Grants, their applications must be deemed of appropriate quality to merit support, but they should not have received funding under the Discovery Grant. The value of the grant is $30,000 over two years, with a maximum yearly allowance of $15,000. For more information, see: NSERC – Discovery Development Grants (nserc-crsng.gc.ca)

Alliance Grants: Alliance Grants encourage university researchers to collaborate with partner organizations from other sectors, generate new knowledge and /or technology and accelerate the application of research results to create benefits for Canada.

Canadian university researchers eligible to receive NSERC funding can apply as individuals or with eligible co-applicants. The funding period is for one to five years and the funding value is $20,000 to $1 million per year. Alliance Grants have a cost-sharing component between project partners. For more information, see: NSERC – Alliance grants (nserc-crsng.gc.ca)

Collaborative Research and Training Experience program (CREATE): CREATE is intended to support the development of innovative training programs to encourage collaborative and integrative approaches to address scientific challenges associated with Canadian research priorities. CREATE aims to facilitate the transition of natural sciences, engineering or interdisciplinary trainee students and postdoctoral researchers to productive employees in the Canadian workforce by fostering the acquisition and development of professional skills.

NSERC-eligible institutions can hold grants, and the applicant must be in an eligible position at the institution and be from an NSERC-supported field. CREATE funding is for up to six years. There are two stages to the application process and there are three streams to which applicants can apply: regular, industrial and international collaboration. Applicants successful at the second stage can receive up to $150,000 in the first year and up to $300,000 for the following five years, for a maximum of $1.65 million over six years: 80% of funds must go to trainee stipends and up to 30% can go to trainees in non-NSE fields. For more information, see: NSERC – Collaborative Research and Training Experience program (nserc-crsng.gc.ca)

CIHR Award Programs and Health Research Training Initiatives

CIHR Doctoral Research Award Priority Announcements. These awards offer additional sources of funding for highly rated applications that are relevant to CIHR research priority areas or mandates. The Doctoral Research Awards are intended to provide recognition and support for students who are pursuing a PhD degree in a health-related field in Canada or abroad.

Funding level per award is generally the same as the CGS D ($35,000 per year, for up to three years). Other awards offered vary by research area. For more information, see: Doctoral Research Award : Fall 2021 Priority Announcement (ResearchNet) and Doctoral Research Awards – Application instructions (cihr-irsc.gc.ca)

Doctoral Foreign Study Awards (DFSA): The DFSAs intend to support students abroad (outside Canada) who are pursuing a doctoral degree in a health-related field. The award provides recognition and funding to students in their academic research career and supports students gaining research experience abroad.

Applicants previously enrolled in a graduate program must have completed no more than 24 months of full-time study in their doctoral program. Applicants enrolled in a joint program and those enrolled directly from a bachelor’s degree to a doctorate must have completed no more than 36 months of full-time study in their doctoral program. A primary supervisor must be identified and affiliated with the institution abroad where the doctoral studies will be conducted. The maximum amount per award is $35,000 per year for up to three years, with a $30,000 stipend and research allowance of $5,000 per annum.^{Footnote 83} For more information, see: Doctoral Research Award : Doctoral Foreign Study Award (DFSA) (ResearchNet)

Master's awards: The master’s awards (Science to Business Program and Dr. James Rossiter Master’s of Public Health) offered additional sources of funding for highly rated applications relevant to CIHR research priority areas or mandates and were intended to provide recognition and support for students in Canada who 1) were pursuing a master’s degree in a health-related field or 2) had a health-related PhD and were pursuing a MBA. Funding levels varied depending on the funding opportunity and an allowance/supplement was also available to students who wished to travel abroad. These awards are no longer offered.

Health Research Training Platform (HRTP): HRTP intends to support the development of interdisciplinary, interjurisdictional and intersectoral research training platforms to attract a diverse cadre of trainees and early career researchers and equip them with the skills required for academic and non-academic careers. This grant aims to support training and mentoring platforms that build capacity in areas of major health issues, scientific opportunities and critical gaps.

The amount of funding is enough to fund approximately 12 grants, with a maximum amount per grant of $400,000 per year for up to six years, for a total of $2,400,000 per grant. This funding opportunity supports a list of training platforms that applicants can apply for. The nominated principal applicant must be an independent researcher or knowledge user and include two other members identified as principal applicants. Applicants collectively must be affiliated with at least three different eligible institutions. The proposed training platform must identify at least eight mentors representing different institutions, disciplines, jurisdictions, sectors and career stages. One member must serve the function of an Equity, Diversity and Inclusion (EDI) Champion. For more information, see: Training Grant: Health Research Training Platform (ResearchNet)

The HRTP was not in the scope of this evaluation as it was a new initiative with funds not yet disbursed.

Health System Impact (HSI) Fellowship: The HSI Fellowship intends to provide doctoral trainees and postdoctoral fellows with an opportunity to apply their research and talents to challenges in health care that are being addressed by health system and related organizations outside the university setting, and to develop professional experience, new skills and networks. This funding opportunity aims to support research areas that focus on health services, health policy and/or health system challenges.

Doctoral applicants must be a trainee and enrolled full-time studying health services and policy research (HSPR) or related fields for their doctoral training at a Canadian eligible institution. Postdoctoral applicants must be a trainee and have obtained their degree in a HSPR or related field. For doctoral trainees, the maximum amount per award is $50,000 for one year, with a $45,000 stipend and a training and research allowance of $5,000. For postdoctoral fellows, the maximum amount per award is $155,000 total for two years, with a $70,000 stipend and a training and research allowance of $7,500 per annum. For more information, see: Health System Impact Fellowship (ResearchNet)

Investigator-Initiated Research

Investigator-initiated research programs provide funding to conduct research in any area related to health aimed at the discovery and application of knowledge. Funding is provided to researchers and academic organizations to conduct research, translate knowledge and build capacity through research training and salaries. Applicants identify and propose the nature and scope of the research and compete for support by demonstrating excellence and the potential impact the research will have on health systems and/or health outcomes.

Descriptions of CIHR’s current IIR programs are provided below.

Project Grant Program: The Project Grant program is designed to capture ideas with the greatest potential for important advances in fundamental or applied health-related knowledge, health care, health systems and/or health outcomes. It intends to support projects proposed and conducted by researchers in all areas of health. The program aims to support a diverse portfolio of health-related research and knowledge translation proposals, promotes collaborations and intends to contribute to the creation and use of health-related knowledge.

Funding levels are proportionate to the requirements of the research proposed, which is expected to vary by research field, approach and scope of proposal activities. Applicants must be an independent researcher or a knowledge user affiliated with a Canadian postsecondary institution and/or its affiliated institution (i.e., hospitals, research institutes and other not-for-profit organizations with a mandate for health research and/or knowledge translation). An Indigenous non-governmental organization or an individual affiliated with an Indigenous non-governmental organization in Canada with a research and/or knowledge translation mandate can apply. For more information on the Project Grant Program, see: Project Grant (ResearchNet)

Foundation Grant Program: The Foundation Grant Program ended in 2019. It intended to provide long-term support for established health research leaders across all health-related disciplines. The grant aimed to provide the flexibility to pursue novel and innovative lines of inquiry and contribute to the creation and application of health-related knowledge.

Grant levels varied by research field, approach and scope of program activities. The duration of the grant was seven years. Applicants had to be mid-career/senior researchers who were independent researchers with a demonstratable track of excellence. For more information, see: Foundation Grant (ResearchNet) and Message from the President: Final decision on the Foundation Grant program (cihr-irsc.gc.ca)

Research in Priority Areas

Priority-driven research programs provide funding for targeted grants and awards aimed at addressing priority areas. The goal of this funding is to advance health knowledge and its application, in specific areas of research identified by CIHR in consultation with other government departments, partners and stakeholders, to improve health systems and/or improve health outcomes in these priority areas. Grants are disbursed to fund research or to provide career or training support.

Descriptions of programs referred to and included in the evaluation are provided below; however, information on other priority-driven programming can be found on CIHR’s website and additional initiatives, led by CIHR Institutes, can be found on the institutes’ websites.

Catalyst Grants: These short-term grants catalyze and support research in priority-driven areas. They often support new and novel ideas and development of preliminary data. The specific research areas or initiatives vary. Eligibility criteria also vary depending on the research areas of focus and funding opportunity objectives. The nominated principal applicant tends to be an independent researcher with an academic appointment at an eligible institution. However, the eligibility can be expanded to include Indigenous non-governmental organizations in Canada and affiliated individuals. Some examples include: HIV/AIDS Community-Based Research, Muskuloskeletal Rehab and ME/Chronic Fatigue Syndrome.

Collaborative Health Research Projects (CHRP) Program: The Collaborative Health Research Projects (CHRP) program was a joint initiative between CIHR, NSERC and SSHRC (the latter through special calls only). CHRP grants support focused on interdisciplinary, collaborative research projects involving any field of the natural sciences or engineering and any field of the health sciences. It ended in 2022.

Funding levels were proportionate to the requirements of the research proposed, which was expected to vary by approach and scope of proposal activities. The proposal had to include at least two principal applicants (including the nominated principal applicant), one from the natural sciences or engineering community, and one from the health sciences community. Both applicants must have held academic appointments at Canadian institutions. The application must have to include trainees, and the application must had included a non-academic knowledge/technology user (KTU) organization with an expressed interest and demonstrable ability to use the end products of the research. Collaborative Health Research Projects - CIHR (cihr-irsc.gc.ca)

Priority Announcement Grants: Priority Announcement (PA) Grants offer additional sources of funding for highly rated applications submitted to other grant funding opportunities (e.g., Project Grant Program) that are relevant to CIHR research priority areas or mandates.

Funding levels vary for each research area and this grant shares eligibility with the Project Grant competition. Applicants can identify up to three PAs for consideration. For more information, see: Project Grant : Spring 2022 Priority Announcement (Specific Research Areas) (ResearchNet) and and Project Grant Program: Application Process - CIHR (cihr-irsc.gc.ca)

Strategy for Patient-Oriented Research (SPOR): SPOR funds patient-oriented research (meaning patients need to be involved in the research projects and processes). SPOR has funded a wide variety of patient-oriented research through two main mechanisms: SPOR Networks and Innovative Clinical Trials (iCTs). For additional details on eligibility, visit the links provided. Strategy for Patient-Oriented Research - CIHR (cihr-irsc.gc.ca).

Summer Program in Aging Host Support: The Summer Program in Aging (SPA) is the Institute of Aging’s flagship training program that intends to provide graduate students and postdoctoral fellows involved in aging research with a program of advanced training. This program brings together research trainees and academic mentors to participate in interactive learning sessions.

Applicants must be a trainee and enrolled full-time or part-time in a master’s or doctoral program or be granted a postdoctoral fellowship at a Canadian university or follow postgraduate clinical training. An academic supervisor must be identified, and the trainee must be attending a SPA Training Program for the first time. For more information, see: Summer Program in Aging (2022) (ResearchNet) and CIHR Institute of Aging Summer Program in Aging 2022 (cihr-irsc.gc.ca)

Team Grants: These larger grants support team-driven research in priority-driven areas. They often promote collaboration and coordination across groups of researchers. The specific research areas or initiatives vary. Eligibility criteria also vary depending on the research areas of focus and funding opportunity objectives. The nominated principal applicant tends to be an independent researcher with an academic appointment at an eligible institution. However, the eligibility can be expanded to include Indigenous non-governmental organizations in Canada or affiliated individuals. Some examples include: HIV/AIDS Comorbidities Prevention, Healthy Living, Canada-UK Artificial Intelligence Initiative, Cannabis Research in Priority Areas.

Training Grant: Indigenous Mentorship Network Program: The Indigenous Mentorship Network Program was sunset in 2019. It intended to support researchers and leaders with a track record in successfully mentoring First Nations, Inuit and Métis trainees and new investigators who had experience in community-based research. It aimed to support trainees and new investigators and increase the number and competitiveness of First Nations, Inuit and Métis researchers through mentoring activities.

The maximum amount per grant was $200,00 per year for up to five years for a total of $1 million per grant. Each grant supported seven provincial nodes and one national/international coordinating node. Applicants were independent researchers with significant lived experience and were appointed at an eligible institution. Other applicants or collaborators could be non-Indigenous, but a majority of principal applicants had to be Indigenous. Full applications had to identify a minimum of five Indigenous mentees. For more information, see: Training Grant : Indigenous Mentorship Network Program (ResearchNet), Indigenous Mentorship Network Program (cihr-irsc.gc.ca), and Update on the CIHR Indigenous Mentorship Network Program (cihr-irsc.gc.ca)

MD/PhD Program Grants: The MD/PhD program grants helped to strengthen the Canadian clinical research endeavour by increasing the number of clinician-investigators. The program ended in 2015, with funding continuing until 2021. Funds were allocated to eligible Canadian universities, which then distributed vouchers to trainees. Principal applicants were nominated and must have been the MD/PhD Program Director at an eligible institution. The funds must have been used by the MD/PhD Program Director to support students who were enrolled in a combined MD/PhD program at their institution. The research supervisors of the supported students should have held operating and/or salary funds obtained through a competitive peer review process. The value of the vouchers was a maximum of $22,000 (trainee stipend of $21,000 per year and a research stipend of $1000 per year, for up to six years).

Appendix D. Methodology

This appendix provides an overview of the methodology for this study. A detailed technical report is also available on request.

As defined in the Treasury Board Secretariat Policy on Results (TBS, 2016), evaluations typically focus on programs, policies, and priorities and examine other units or themes as determined by user needs. The design of this evaluation was developed in consultation with agency staff and representatives from stakeholder organizations.^{Footnote 84} Three focus areas were identified: student access, experience, and trajectory. These focus areas were the basis for the following evaluation questions:

• Who does the portfolio reach? This question focused on student access to funding in the portfolio. It considered whether the agencies have the right mix of FOs to reach a diverse array of graduate students, considering the agencies’ EDI commitment to access for all trainees.
• How does this portfolio contribute to the research training environment? This question focused on student training experience and the agencies’ influence on the research training environment.
• Does this funding influence student career trajectories? This question focused on how students’ career goals and early career outcomes relate to their funding and training opportunities.
• What should the agencies consider in the context of changes in the training environment and/or student experience due to COVID-19? The impact of the pandemic on the training environment was also considered as part of this evaluation.

These questions address relevance and effectiveness of the funding as required under TBS Policy.

Design. The evaluation used a mixed methods approach implemented in phases. Results from separate lines of evidence were integrated at each phase and the results used in decisions for the next phase (e.g., instrument development, sampling). Milestone reports following each phase were presented to the Evaluation’s Advisory Committee and senior management at the three agencies. A final synthesis produced the findings reported below.

The evaluation applied an EDI lens throughout all stages.^{Footnote 85} While no single funding opportunity (FO) will reach all students, the FOs in the portfolio may work together to provide a range of opportunities for students to access support and for the funding to contribute to a rich and inclusive training experience, in alignment with the agencies' EDI commitments (NSERC, 2022a, 2022b). The EDI lens informed decisions about the evaluation's priority areas of focus, its questions and indicators, data sources and analytical approach. This ensured explicit attention to diversity and inclusion and potential barriers and enablers for underrepresented groups.^{Footnote 86}

The evaluation incorporated six lines of evidence. The lines of evidence were chosen to provide multiple perspectives on portfolio funding and its role in the funding system. Primary data were collected through key informant interviews, case studies, focus groups and surveys. The evaluation also made extensive use of secondary data, including agency administrative data, external stakeholder reports and survey data from external organizations, summarized in Table 4.

Scope. As this evaluation was designed to support the agencies’ future strategy for training, the focus was on funding that the agencies offered at the time the evaluation began (2020/21) and may wish to improve or adapt going forward. For NSERC and SSHRC, any funding that had been awarded as a single one-time grant or which had ended^{Footnote 91} was excluded. A similar approach was taken by CIHR, although CIHR’s scope included some relevant funding through their Research in Priority Areas (RPA) and Investigator Initiated Research Programs (IIR) that is no longer offered as of 2020-21.

An additional set of FOs exist that are relevant to graduate student training but were not studied in depth. They include FOs that had been very recently evaluated such as the CGS Vanier (CIHR, 2020b), FOs delivered by agency partners (e.g., MITACS Accelerate Joint Funding), newly implemented FOs for which data were not yet available (e.g., CIHR’s Collaborative Health Research Training Platform) and funding not intended for graduate students. This funding is important to acknowledge for the potential impact on student trajectory, despite not being examined in depth here. For example, funding for undergraduate students and postdoctoral fellows, and agency funding for infrastructure at universities.^{Footnote 92}

The evaluation included a maximum of seven years of data (2014-21) for the FOs in its scope.^{Footnote 93} However, FOs in the current portfolio have been introduced at different times, as the agencies review and adapt their funding regularly. For example, the Canada Graduate Scholarships (direct stream) were introduced in 2003, but NSERC’s Alliance Grants (indirect stream) were introduced in 2019. Additional information about the timeframe is provided below.

Important methodological considerations

Key considerations for interpreting the findings from this study are described next.

Variation in program delivery. Much of the control over student access and training experience rests largely outside the agencies. The agencies’ FOs in this portfolio are delivered in a decentralized way, with substantive decisions made at the institution or grant holder/supervisor level. This presents challenges for tracing the funding’s contribution to student experience and trajectory, relative to other factors. The shared delivery of the portfolio also introduces the potential for wide variation in program implementation, which can make outcomes of the funding variable.

Funding and application status. Students may have participated in multiple agency funding opportunities over the course of their studies. Some students apply for agency awards multiple times, within and across degrees. Some also participate in both funding streams. Additionally, students may be receiving funding from other, non-agency sources. This results in potential interactions among sources of funding and an inability to separate students into two discrete funded and not funded comparison groups. Participants in this portfolio fall into the following five categories:

1. Direct awards - unsuccessful applicants. Applicants for direct support who did not receive an agency scholarship/fellowship at any point between competition years 2014 and 2021^{Footnote 94} and were not identified as recipients of agency indirect support.^{Footnote 95}
2. Direct awards - recipients. Recipients of at least one scholarship/fellowship from the agencies between competition years 2014 and 2021, who were not identified as recipients of agency indirect support.
3. Indirectly funded and award recipients. Recipients of at least one scholarship/fellowship from the agencies between competition years 2014 and 2021, who were also identified as having received indirect support from the agencies.
4. Indirectly funded and unsuccessful applicant. Individuals who received indirect support from the agencies between 2020 and 2021 who also applied for but did not receive an agency award between 2014-21.
5. Indirectly funded only (non-applicants). Students who received indirect support between 2020 and 2021 and had not applied for agency direct awards.^{Footnote 96}

Timeframe. The seven-year timeframe for the evaluation was important to provide sufficient sample sizes for underrepresented groups, whose voice is important to this evaluation. It also allowed for past participants who are now in early career to contribute perspectives. However, differences in available data for sampling^{Footnote 97} meant that, while the timeframes overlap for students in both direct and indirect streams, the direct stream data included more students who participated earlier in the timeframe.⁹⁸ The results for direct stream participants include students who applied for or received an award between 2014 to 2021.^{Footnote 99} The end date of recipient funding varies for different awards. Indirect stream participants were receiving funding in 2020-21, and the start date of their participation varies, with some having participated in the indirect stream over the entire 2014-21 period.

It was not possible to group students into equivalent timeframes for cohort comparisons. This is due to varying duration and multiple instances of participation by students within both direct and indirect funding streams. To validate comparability of the data between the two streams, the direct stream data were analyzed separately for each of the seven competition years from 2014-21. Any notable differences in early to late period results, or indications of change over time, are reported below.

Population size estimates from external surveys are reported with the results below to provide context to portfolio participation data. The timeframes for population estimates also vary, but in most cases are from 2019 or earlier. The source and year for all population estimates are provided.

Survey sample for doctoral awards. Direct awards at the doctoral level are adjudicated at two levels for most applicants, first within institutions and then in a national level competition. Data for all unsuccessful applicants to SSHRC, at both levels of competition, were available for sampling. However, while NSERC and CIHR data were available for applicants who had reached the national competition level, which is the level of competition managed by the agencies, only partial data were available for unsuccessful applicants at the institutional level of competition. As a result, the student survey sample differed between agencies for unsuccessful doctoral award applicants.^{Footnote 100} SSHRC results are reported above for national competition applicants separately, to be comparable to NSERC and CIHR. Any differences found in responses between national competition and institution-level applicants for SSHRC are also reported. Where available, administrative data for NSERC and CIHR doctoral applicants who did not reach the national competition are reported where pertinent.

Analysis for nonresponse bias found the student survey direct stream respondent sample to be comparable to the agencies’ direct awards population, based on the population parameters available in administrative data. No indication of nonresponse bias was identified, with a few exceptions: relatively lower response rate for older cohorts (pre-2018) and SSHRC doctoral applicants who did not reach the national competition level; and higher representation of respondents self-identifying with a disability or LGBTQ2+ communities.

Survey sample for students participating in the indirect stream. The indirect stream student survey sample was developed via a multi-stage approach. Agency grant holders (n=31,306) were surveyed first to identify students financially supported by indirect stream grants between 2020 and 2021. Overall response was 36%, including 21% of co-applicants and 45% of principal grant holders, i.e., PI or NPI. All students meeting inclusion criteria^{Footnote 101} who were identified by grant holders were then invited to participate in the student survey (n=24,406). Before starting the survey, students were asked for their informed consent to participate. Contact information collected through the grant-holder survey for students who did not consent to participate was discarded. Except for the varying response rate by agency,^{Footnote 102} analysis of response to the grant-holder survey found no indication of non-response bias based on factors available in administrative data related to grant and grant-holder population. The agencies had little prior information about the students funded through the indirect stream, however. Therefore, analysis for nonresponse bias by indirectly funded students was limited. For these reasons, the generalizability of findings to indirectly funded students beyond the sample may be limited. Caution is recommended.

Given the purpose of this evaluation, findings have been reported at portfolio level, with breakdowns at lower levels (e.g., funding stream, funding agency, demographic subgroup). The report provides descriptive statistics for both direct and indirect stream respondents to the student survey at all levels. The use of inferential statistics was limited to direct stream participant data only. This is due to the partial sample frame^{Footnote 103} for indirectly funded students and limited ability to assess representativeness of the sample (i.e., nonresponse bias). Descriptive statistics and contextual information are provided to support interpretation in all cases because, due to the large size of the sample, statistical significance does not necessarily mean a finding is meaningful.

The COVID-19 pandemic. The surveys, interviews and case studies for this evaluation were conducted in fall 2021 and winter 2022. Response rates to the survey and interview/case study recruitment are similar to prepandemic evaluations for these programs. However, the pandemic may have affected the nature of students’ training experiences in 2020-21. Questions about the impact of the pandemic on students were included in all data collection activities. This created intentional space to discuss the pandemic and its impacts with participants. Importantly, the findings about the impact of the pandemic can inform interpretation of findings for questions about student access and experience, including the extent to which these may reflect unusual conditions.

Methodological and analytical considerations for equity, diversity and inclusion (EDI)

The evaluation applied an EDI lens throughout all stages. The EDI lens informed decisions about the evaluation's priority areas of focus, its questions and indicators, data sources and analytical approach. This ensured explicit attention to diversity and inclusion and potential barriers and enablers for underrepresented groups. Underrepresented groups include, but are not limited, to underrepresented gender identities, Indigenous Peoples (First Nations, Inuit and Métis), persons with disabilities, members of visible minority/racialized groups and members of LGBTQ2+ communities (NSERC, 2022a). Due to the focus on graduate students, this evaluation also considered other groups such as first-generation students, students with dependents, mature students, as well as geographic and linguistic groups.

EDI frameworks and methodological approaches are developing rapidly (Bauer et al., 2021), and this is reflected in changes to the Government of Canada's gender-based analysis (GBA+) approach, which has been adapted to seek to identify intersecting identity factors that shape individuals' experiences accessing, for example, government programs (WAGE, 2022). The importance of intersectionality to studies of social programs is well understood. As described by Bauer et al (2021), it is based in the understanding that: “human experience is jointly shaped by multiple social positions (e.g., race, gender), and cannot be adequately understood by considering social positions independently” (p. 1). Approaching identity in terms of single category membership or treating multiple group membership in a purely additive way in relation to experience or outcomes, is not only oversimplistic but can lead to misinterpretations and misrepresentation of individuals and communities (see, e.g., Else-Quest & Hyde, 2016).

At a technical level, however, methods for intersectional analysis are still developing, particularly for studies that incorporate quantitative data. Alternatives to conventional approaches, at an operational level, are still emerging (see e.g., Bauer et al., 2021). For this evaluation, the qualitative components were used to explore where intersecting group membership may interface with program structures to limit opportunities for students. A fuller intra-categorical analysis drawing on both qualitative and quantitative data was identified as a viable option in terms of available data (McCall, 2005), and important, but was not feasible for all groups within the timeframe and resources for this phase of the evaluation. As this evaluation has been designed to be a longitudinal study, with future phases, a technical protocol will be prepared for secondary, in-depth use of the existing dataset for the next phase.

For this phase, EDI-related analysis of quantitative primary (e.g., survey) and secondary (e.g., administrative) data was based on several considerations. They included availability of data, i.e., sufficiency of sample size for program participants and availability of external population data estimates; recognition of the limitations and risks of reporting disaggregated results by individual subgroups; and resource constraints. Data on representation of underrepresented groups among applicants and recipients of agency funding for graduate students were disaggregated by the identity factors listed above. This disaggregation of EDI data allowed for assessing the extent to which the representation of each equity-seeking group reflects its representation among the graduate student population in Canada (estimates for which were based on external survey and census data). Disaggregation of data was conducted for other analysis of primary and secondary data, to assess the extent to which the selection process is equitable (e.g., comparing success rates of applicants from various groups; self-reported ratings of selection process). However, as outlined above, an important limitation of this analytical approach is that it is univariate and does not take into consideration intersectionality of multiple identity factors.

In recognition of the limitations and risks noted above, disaggregated results by individual subgroups were reported only for the first evaluation question (student access to funding). For other questions (student experience and trajectory), disaggregated results related to key findings were assessed, but reported only where differences in outcome by group were found and had implications for conclusions or recommendations. For some of the student survey questions (i.e., those presented only to smaller sets of respondents, such as those who had graduated), breakdowns of results by identity factors were not feasible due to the smaller sample sizes.

The qualitative components of the evaluation provided different opportunities. Self-identified group membership was not incorporated in participant selection for case study or other key informant interviews for ethical reasons. However, interview questions about potential barriers and enablers related to programs in the portfolio were designed with limited structure to broaden response and avoid incorporating assumptions about social categories. The student survey instrument was designed to include seven open comment questions, and EDI considerations were factored into analysis of these data. The evaluation team relied on primary and secondary self-identification data to ensure that students from equity groups were represented in selection of cases for analysis. Additional resources were applied to enable inductive analysis of a larger set of cases to ensure inclusion and enable saturation. Finally, quantitative data were revisited to further explore emerging findings.

Overall, intersectional analysis was limited and exploratory, but yielded findings related to structural factors that led to recommendations. As noted above, fuller analyses of these data with an intersectional lens are intended for the subsequent phase of the study.

Date modified:

2024-06-10