Evaluation of the Canada First Research Excellence Program (CFREF)

Evaluation of the CFREF Program
January 2021

(PDF, 908 KB)

The Honourable François-Philippe Champagne, P.C., M.P.
Minister of Innovation, Science and Industry

© Her Majesty the Queen in Right of Canada,
represented by the Minister of Industry, 2021

Cat. No. CR1-18E-PDF
ISSN 2563-9382

List of Acronyms

• CAUT – Canadian Association of University Teachers
• CERC – Canada Excellence Research Chairs
• CFI – Canada Foundation for Innovation
• CFREF – Canada First Research Excellence Fund
• CIHR – Canadian Institutes of Health Research
• CRC – Canada Research Chair
• ECR – Early career researchers
• EDI – Equity, diversity and inclusion
• HQP – Highly qualified personnel
• ISED – Innovation, Science and Economic Development Canada
• LOI – Letter of Intent
• NCE – Networks of Centres of Excellence
• NFRF – New Frontiers in Research Fund
• NSERC – Natural Sciences and Engineering Research Council of Canada
• PMP – Performance Measurement Plan
• PSIS – Postsecondary Student Information System
• R&D – Research and development
• SPFR – Survey of Postsecondary Faculty and Researchers
• SSHRC – Social Sciences and Humanities Research Council of Canada
• STEM – Science, technology, engineering and mathematics
• TIPS – Tri-agency Institutional Programs Secretariat
• UCASS – University and College Academic Staff System

List of Tables

• Table 1. Distribution of CFREF applicants and recipients by size of host institution for the first and second CFREF competitions
• Table 2. Aggregate seven-year projections and actual cumulative expenditures as of 2018-19 for both Competition 1 and Competition 2 (n=18)
• Table 3. Detailed breakdown of seven-year projections and actual cumulative expenditures of CFREF funds for compensation-related expenses (salaries and stipends, including benefits), as of 2018-19 for Competition 1 grantees
• Table 4. Level of funding and distribution of partners by sector and location, as of March 2019
• Table 5. Distribution of participants among Competition 1 and Competition 2 CFREF grants from 2015 to 2019
• Table 6. Distribution of new faculty recruited by Competition 1 institutions
• Table 7. Distribution of new HQP recruited by Competition 1 institutions
• Table 8. Distribution of partners among CFREF-funded institutions
• Table 9. Distribution of collaborators among CFREF-funded institutions
• Table 10. CFREF program-level operating expenditure and efficiency ratios

List of Figures

• Figure 1: Distribution of CFREF grants (competitions 1 and 2) based on their predominant priority research area
• Figure 2: Range of financial support provided to faculty members involved in CFREF grants
• Figure 3: The proportion of CFREF researchers (n=239) who expressed concerns about the level of satisfaction with the competitive process used to allocate research funds
• Figure 4: CFREF funding and leveraged support committed by the funded institutions and their partners for the seven-year grant period (as of the end of the 2018-19 fiscal year)
• Figure 5: Competition 1 leveraged funding secured and not yet secured from lead institutions and partners, based on their total commitment for the seven-year grant period, as of March 2019 ($ million)
• Figure 6: Competition 2 leveraged funding secured and not yet secured from funded institutions and partners, based on their total commitment for the seven-year grant period, as of March 2019 ($ million)
• Figure 7: Funding awarded by granting agencies to Competition 1 institutions from the time of drafting the proposal until the mid-term of the grant ($ million)
• Figure 8: Type/role of faculty members involved in Competition 1 grants (N=431)
• Figure 9: Comparison of the representation of the four designated groups among CFREF participants with the representation of the four designated groups in Canada’s overall population and Canada’s labour market
• Figure 10: Representation of individuals from the four designated groups among CFREF faculty compared to full-time professors in Canadian universities
• Figure 11: Representation of women among full-time faculty at Canadian universities by subject area
• Figure 12: Comparison of the representation of the four designated groups among CFREF HQP and the postsecondary sector in Canada
• Figure 13: Distribution of partners by sector, as of March 2019
• Figure 14: Distribution of collaborators by sector

Executive Summary

As the program was launched in 2014, this constitutes the first evaluation of CFREF and covers its initial four fiscal years of operations, from 2015-16 to 2018-19. The primary purpose of the evaluation was to provide an assessment of the relevance and performance of the CFREF program, as well as aspects of design and delivery. The evaluation had a particular focus on immediate outcomes of the first five grants awarded, as it was conducted four years into the delivery of the CFREF program; as such, it was too early to assess intermediate and longer-term outcomes and impacts of these grants and for the program as a whole. Similarly, it was too early to assess longer-term expected results of investing at the institutional level, or to conclude if the program’s focus on funding at the institutional level confers specific advantages or disadvantages compared to funding at the researcher or project level. Nevertheless, based on the information collected, the funded grants have largely met immediate outcomes and demonstrated progress towards achieving intermediate outcomes.

The evaluation involved multiple lines of evidence, including reviews of program documents, relevant literature, and CFREF administrative data (comprising annual progress, mid-term and financial reports from all grantees, as well as the final results of the mid-term peer-review for the five Competition 1 grantees), case studies with the five Competition 1 grantees, interviews with key informants and a web-based survey of CFREF core team members from all 18 grants. The methodology for this evaluation presented some limitations, but these limitations did not prevent the evaluation questions from being adequately addressed.

Relevance of the CFREF program and alignment with government priorities

CFREF occupies a unique niche in the Canadian funding landscape as it is the only large-scale funding program that is designed to support the implementation of scientific and institutional strategies that allow grantees to strengthen their institutional position in a specific field of research. The evaluation concludes that CFREF continues to be relevant, as it provides the government with a unique vehicle for strategically investing in priority research areas which have the potential to create long-term economic advantages for Canada.

The CFREF program is well aligned with government priorities on innovation, talent recruitment, and equity, diversity and inclusion (EDI). The evaluation found that grantees have made some progress in implementing their EDI plans, which is a requirement of the program, to ensure that individuals from the four designated groups (women, visible minorities, Indigenous Peoples and persons with disabilities) have an equal opportunity to participate in and benefit from the program. However, despite efforts to date, program self-identification data indicate that there is an overall underrepresentation of individuals from the four designated groups within the program, particularly among Indigenous Peoples (currently 0.5% of participants) and persons with disabilities (currently 2% of participants). Grantees are aware that this is an important priority for both the agencies and the federal government, and many acknowledge that there is still work to be done to improve diversity in their CFREF teams and governance structures.

Given that support for early career researchers (ECRs) was first introduced as a priority by the Government of Canada in 2018, three years after the CFREF program was launched, this priority did not influence the program’s design at that time. Currently, TIPS collects information on the number of ECRs involved in the initiatives at mid-term, which suggests that the program’s contribution to this priority is of interest to management. However, CFREF’s objectives and expectations of grantees as it relates to supporting ECRs have not been clearly defined in the program information.

The implementation of CFREF

The evaluation has not identified gaps or shortcomings that would raise reasonable concerns about the grantees’ governance structures or their capacity to adequately manage their grants. The flexibility that CFREF offers grantees to build their own governance structure was identified as a key strength of the program by many key informants. The strategic focus appears to evolve over time and the range of funding allocation mechanisms used by grantees is fairly traditional (e.g., competitive processes), but the unifying framework of a common research program distinguishes CFREF grants from other funding the institutions and researchers receive. Grantees have, however, experienced some challenges and delays in the start-up phase, which are common for large-scale funding programs, and have led to a need for some grantees to seek and receive no-cost grant term extensions of two years. At roughly mid-way through their funding, grantees had spent 23% of the $1.2 billion awarded for their grants. As such, the use of funds over time should be carefully monitored going forward, particularly in light of the fact that the current COVID-19 pandemic may cause additional delays.

Participants, partnerships, collaborations and infrastructure

At the time of the evaluation, CFREF-funded activities had engaged more than 6,700 individuals occupying various research or support functions, predominately graduate students (36%), faculty members (23%) and postdoctoral fellows (13%). CFREF participants identified several benefits to participating in the grant, including access to an enhanced interdisciplinary research and training environment, state-of-the-art research facilities and equipment, and complementary training programs that develop HQPs’ non-academic skills (e.g., knowledge translation and commercialization) and employability.

Grantees have engaged more than 600 partners and close to 1,500 collaborators, at the national and international levels. The exact contribution of CFREF in allowing these partnerships and collaborations to emerge or expand cannot be measured precisely, but evaluation findings indicate that receiving grants of the magnitude of CFREF has facilitated this outcome. As of the end of the 2018-19 fiscal year, one-half of the $1.3 billion committed by the funded institutions and their partners for the seven-year period covered by the grants had been secured. In addition, these partnerships and collaborations have provided grantees with growing visibility and recognition at national and international levels, as well as access to a wider range of infrastructures, equipment and expertise, both from a scientific and commercialization perspective.

Over the seven-year grant term, the 17 grantees have projected to spend a combined total of $255 million on research facilities, equipment and supplies. The CFI has played a critical role in providing complementary support to ensure that the required infrastructures are available to conduct the funded research. Data from Competition 1 grantees show that these five institutions had secured $71 million since drafting the CFREF proposals up until mid-term.

Program design and delivery

The processes relating to the first two competitions were fairly challenging for all involved. Given the large scale of the funding provided by CFREF and the significant latitude for applicants in terms of determining their scientific and institutional strategies, there were many uncertainties surrounding the application process, and both successful and unsuccessful applicants had to seek more guidance and clarification from TIPS than they had received. As for the ongoing implementation of the grants, grantees would generally appreciate having more sustained communications and interactions with TIPS to ensure that they are proceeding in accordance with the expectations of the funding agencies.

The evaluation also identified the strengths and limitations of the current monitoring and reporting activities undertaken by TIPS and CFREF grantees. The performance information collected by TIPS through the annual and the mid-term reports was valuable for this evaluation and provided a good overall understanding of the grantees’ activities and the progress made by the program toward its immediate outcomes. Nonetheless, some important areas for improvement were identified, including: reviewing the reporting templates in order to ensure greater consistency and enhance quality of data collected though these reports, clarifying the purpose and intended use of performance data collected through the PMPs, encouraging grantees to clearly articulate a long-term vision for what they want to accomplish through their grant, and that TIPS explore the possibility of instituting an end-of-grant report. This would allow grantees to document their overall experience with their CFREF grant, the results achieved, the expected legacy or long-term impacts of the grant, key challenges and lessons learned. This reporting would also facilitate TIPS’ efforts to document and report on program outcomes and would provide evidence for future evaluations.

Recommendations

Although it was too early in the program’s lifecycle to assess the longer-term expected results of investing at the institutional level, the program remains relevant, has largely met its immediate outcomes (i.e., governance structures and funding allocation processes within institutions, partnerships, collaboration, attraction/retention of teams) and demonstrated progress toward some of its intermediate outcomes (i.e., infrastructures, training environments). This conclusion is supported both by the data collected for this evaluation and the results from the mid-term review of the first five grants. It should be noted, however, that there remain some questions regarding how the transformational changes brought about by the CFREFs will be sustained. The manner in which grantees described their plans for sustaining the transformative changes largely focused on what other funding would be sought to allow them to maintain the momentum of their research activities once their CFREF grants ended. Although grantees described some activities and early outcomes that are indicative of legacy and point to long-term institutional impacts of the program (e.g., new faculty positions created in areas of the CFREF and enhancements to the training environments), the overall results from the mid-term review and the evaluation suggest that securing funding for sustaining the transformational changes brought by the CFREFs could be a challenge following the end of the granting period.

An analysis of cost-efficiency data suggests that the CFREF program has been delivered by TIPS in a very cost-efficient manner to date, however, evaluation findings suggest that TIPS’ administrative costs of delivering the program may be too low for supporting effective implementation and monitoring. Specifically, grantees and applicants identified some challenges with respect to design and delivery of the CFREF program, some of which could be mitigated by improving communications between TIPS and grantees/applicants. The evaluation also identified strengths and limitations of the current monitoring, reporting and performance measurement activities. Based on these conclusions, the evaluation offers the following recommendations to improve the CFREF program.

1.0 Introduction

This document constitutes the final report of the evaluation of the Canada First Research Excellence Fund (CFREF). SSHRC evaluators and an evaluation consulting firm (PRA Inc.) collaborated to design and implement this evaluation. As the program was launched in 2014, this constitutes the first evaluation of CFREF and covers its initial four fiscal years of operations, from 2015-16 to 2018-19. The evaluation was conducted in accordance with the federal government’s Policy on Results (2016) and section 42.1(1) of the Financial Administration Act, which requires each ongoing federal program of grants and contributions to be evaluated every five years with respect to its relevance and effectiveness.

The following subsections included in this introduction provide a brief overview of CFREF, of the purpose and scope of the evaluation, and a summary of the methodology used to evaluate CFREF. A more detailed description of the methodology is included in Appendix B.

1.1 Overview of CFREF

As part of its Economic Action Plan, the federal government announced the establishment of CFREF in its February 2014 budget. At that time, it set aside $1.5 billion over 10 years to support this new program, which was expected to strengthen the capacity of world-class postsecondary institutions in Canada to recruit leading researchers, secure promising partnerships, and advance breakthrough discoveries. The federal government’s ultimate goal was to “help Canadian post-secondary institutions excel globally in research areas that create long-term economic advantages for Canada” (Government of Canada, 2014b, p. 115).

At the time of the evaluation, CFREF had funded 18 seven-year grants through a competitive peer-review process to 17 postsecondary institutions (i.e., 17 grantees). These grants support the implementation of institutional and scientific strategies in research areas where these institutions have already demonstrated strength and leadership. Described succinctly, the “institutional strategy” refers to the overall vision and approach that the institution as a whole is proposing to integrate the CFREF grant into its existing governance and operations, while the “scientific strategy” refers to the actual research program that the institution is proposing to undertake, including the engagement of partners and collaborators and the research program’s anticipated impact.

Institutions that are awarded CFREF grants may use the funding to cover both direct and indirect costs of research^{Footnote 1}:

• Direct costs may include, among other things, expenses related to compensation (salaries, scholarships or stipends) for students, postdoctoral fellows, new faculty members, technicians and other research professionals, as applicable. They may also include expenses related to some equipment (typically valued at less than $300,000) and supplies, and the dissemination of research activities. Finally, direct costs may include seed funding to conduct peer-reviewed competitions that are aligned with the scientific strategy of the institution.
• Indirect costs may include, among other things, expenses related to the renovation and maintenance of research facilities, research resources (e.g., library holdings), intellectual property and licensing, and the management and administration of the grant. Indirect costs cannot exceed 25% of the total grant amount.

The Tri-agency Institutional Programs Secretariat (TIPS) is responsible for the administration of the program, including managing the peer-review process of proposals by panel members and the release of funds to CFREF grantees, ensuring ongoing eligibility of institutions to receive funding, as well as ongoing monitoring and management of funding agreements signed with each institutional grantee. Thus far, TIPS has managed two competitions for CFREF grants, worth $1.25 billion:

• The results of Competition 1 were announced in July 2015, when $350 million was awarded across five grants. The value of each grant ranges between $33.5 million and $114 million.
• The results of Competition 2 were announced in September 2016, when $900 million was awarded across 13 grants. The value of each grant ranges between $33.3 million and $93.7 million.

For each of the two competitions, funding applications were expected to address the research priority areas outlined in the 2014 Science, Technology, and Innovation Strategy, namely: environment and agriculture; health and life sciences; natural resources and energy; information and communication technologies; and advanced manufacturing (Government of Canada, 2014a, p. 20). In 2019, TIPS announced that the third competition for CFREF funding is expected to be launched in 2021-22 (Government of Canada, 2014c).

1.2 Evaluation purpose and scope

The primary purpose of the evaluation was to provide senior management from the tri-agencies―the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Social Sciences and Humanities Research Council of Canada (SSHRC)―with an assessment of the relevance and immediate outcomes of CFREF as per the program’s logic model (included in Appendix A). As it was conducted four years into the delivery of the CFREF program, it was too early to assess the longer-term expected results of investing at the institutional level, or to conclude if the program’s focus on funding at the institutional level confers specific advantages or disadvantages compared to funding at the researcher or project level. Aspects of design and delivery were looked at to help support the ongoing management of the program. More precisely, the evaluation addressed the following five questions:

1. To what extent does CFREF continue to address a unique need and align with government priorities?
2. How, and to what extent, have institutions implemented structures and processes for prioritizing funding toward research in CFREF priority research areas?
3. To what extent has high-caliber, diverse and interdisciplinary research talent been attracted, retained and trained?
4. To what extent have funded institutions created or strengthened partnerships, collaborations and infrastructure to enhance research capacity?
5. To what extent is the design and delivery of CFREF effective and efficient?

In accordance with the Directive on Results, the evaluation also took into account government-wide policy considerations such as equity, diversity and inclusion (EDI). The evaluation has documented the experience of grantees in integrating EDI considerations in the implementation of their activities, particularly as it relates to recruiting new participants and establishing collaborations.

At the time of data collection for this evaluation, Competition 1 grantees had completed approximately four years of activities, while Competition 2 grantees had completed approximately three years of their seven-year grant term. Although it is too early in the life of these grants to assess intermediate and long-term outcomes, all 18 grants were included in this evaluation in order to build our understanding of how CFREF grants have been administered to date, what progress has been made towards short-term goals and objectives, and how the CFREF program as a whole is unfolding and being received within institutions.

Finally, it should be noted that one of the key components of the monitoring activities related to CFREF is a mid-term review of each individual grant, which is expected to occur by the end of the fourth year of the grant. This process, managed by TIPS, starts with a mid-term report that is prepared by the grantee. This report is then reviewed by external experts who provide written assessments. On that basis, a multidisciplinary review panel that includes individuals with expertise on the subject matter of the CFREF grant conducts a site visit and provides its assessment to the Mid-term Board that, in turn, provides recommendations that are subject to approval by the CFREF Steering Committee.^{Footnote 2 Footnote 3} At the time of this report, the mid-term review for all five Competition 1 grants had been completed. Ultimately, the evaluation and the mid-term reviews played complementary roles, with the former addressing the relevance and performance of the CFREF program as a whole, and the latter providing assessments of each grant to date, with latitude to make recommendations for improving scientific and implementation strategies.

1.3 Evaluation methodology and limitations

The evaluation involved multiple lines of evidence, including a review of program documents and some relevant literature, a review of CFREF administrative data (comprising annual progress and financial reports from all grantees, as well as the final results of the mid-term peer review for the first five grants^{Footnote 4}), case studies of the five Competition 1 grantees^{Footnote 5}, interviews with key informants (n=62 interviews involving 92 individuals)^{Footnote 6} and a web-based survey of CFREF core team members from all 18 grants.^{Footnote 7} The overall response rate for the survey was 20% (n=1,144/5,671), but varied by institution (between 6% and 33%) and it was slightly higher among faculty (27%, n=410/1,505). A detailed description of the methodology is included in Appendix B.

The methodology for this evaluation presented some limitations described in this subsection, but these limitations did not prevent all evaluation questions from being adequately addressed. First, although unsuccessful institutional applicants to the CFREF program were interviewed as part of this evaluation, researchers at recipient institutions who were unsuccessful in receiving CFREF funding, and researchers who work in similar research areas (as those of the CFREF grants) at universities that did not receive a CFREF grant, were not interviewed as part of this evaluation. This reflects the fact that the evaluation focussed on CFREF as a whole and was not meant to evaluate each grant individually. As such, the primary goal was to obtain the views of researchers funded through CFREF, and also receive some insights from applicants to the CFREF program who were not successful.

Second, some of the administrative data reviewed as part of this evaluation was not reported consistently across CFREF grantees (e.g., timeframe of reported data). In addition, at the time of the evaluation, the mid-term reports were still not submitted by Competition 2 grantees. As such, more detailed administrative data was available for Competition 1 grantees. In order to address these issues, administrative data was aggregated across all grantees only when it was feasible to do so.

Third, the response rate of the survey of CFREF participants varied among CFREF grants (between 13% and 33% except for one grant where the response rate was 6%). Despite these variations, a sufficient number of respondents from both faculty and HQP was obtained to allow for the survey results to be considered representatives of all 18 grants. Moreover, the sample of survey recipients (i.e., lists of CFREF participants) was validated to ensure that the responses of CFREF participants with only peripheral involvement in the grant were excluded from the dataset.

2.0 The relevance of CFREF

This section provides a brief assessment of how the relevance of CFREF has evolved since the program was launched in 2014. This includes looking at CFREF’s niche in the current ecosystem of federal research funding and how CFREF has been positioned in relation to government priorities. The brief exploration of these issues was based on a literature review complemented with perceptions of interviewees.

2.1 CFREF’s niche in the current ecosystem of federal research funding

Any component of an ecosystem of research funding is bound to co-exist and interact with other programs that may support, differently prioritize, duplicate or complement what it is attempting to achieve. As such, the predominance of the concept of an ecosystem when assessing research funding reflects the inescapable fact that any level of excellence in research is never the result of a single initiative. It is rather the achievement of the ecosystem as a whole. In this context, the primary goal when assessing any research funding program, including CFREF, is to better understand its relative niche and contribution, if any, toward this goal of research excellence.

The rationale for introducing CFREF

Research as engine of growth

As noted in the introduction, the federal government initially included CFREF as part of its Economic Action Plan in 2014, and the rationale for the investment relied heavily on the program’s anticipated ability to create long-term economic advantages for Canada. The fact that the funding was directed toward five priority research areas with high levels of commercialization potential, and would involve an array of partners, both public and private, further illustrates this vision.

It has, in fact, long been recognized that research and development (R&D) and the innovation it facilitates are an essential requisite for sustained economic growth (Canada’s Fundamental Science Review Panel, 2017, p. 21; Research, Technology & Development Topical Interest Group, 2015, p. 13). Despite its relatively small population base, Canada has historically positioned itself favourably when it comes to knowledge creation. As of 2014, Canada was among the top 10 producers of scientific publications, alongside countries with far larger populations and R&D investments such as the United States, China, Germany, the United Kingdom and France (Council of Canadian Academies, 2018, p. 37).^{Footnote 8}

Yet, at the time that CFREF was launched, Canada was starting to lag at the international level in relation to its capacity to remain competitive in research and development (R&D) investments, knowledge creation, and its capacity to attract and retain world-leading researchers (Canada’s Fundamental Science Review Panel, 2017; Council of Canadian Academies, 2018; OECD, 2018; Scherer, 2014). As it announced the creation of CFREF, the federal government emphasized that the level of international competition for the best minds, partnership opportunities and breakthrough discoveries implied was such that Canada could not afford to be complacent and had to support its world-class institutions to advance their greatest strengths and maintain their competitiveness on the global stage (Government of Canada, 2014b, p. 115).

Investing at the institutional level

Arguably, one of the most distinguishing features of CFREF is its focus on supporting large-scale strategic research initiatives led by institutions, rather than supporting individual researchers or research projects. CFREF further narrows its scope by targeting postsecondary institutions that have already distinguished themselves as leaders in specific fields of research. As such, the purpose of CFREF is not to enable institutions to build foundational capacity in a field of research. Instead, the aim of the program is to further support well-established institutions in a specific field to significantly expand their pre-strong pre-existing capacity to undertake research, attract top minds, and build partnerships. Some stakeholders, echoing an issue raised by the Fundamental Science Review^{Footnote 9}, questioned whether the federal government’s decision to concentrate resources on a small number of grants represents the best funding model, compared to allocating smaller amounts of money to a larger pool of researchers and/or institutions, and expressed concern that this approach may lead to an overconcentration of resources in a subset of institutions. This view was expressed particularly by those applicants who had not been successful in securing CFREF funding. Other interviewees, however, (both funded and unfunded) lauded this approach, saying that it shows strong commitment by the federal government to invest in priority areas and raise the international profile of Canadian postsecondary institutions.

As for the types of institutions that may access CFREF support, administrative data to date demonstrates that participating institutions and grantees vary in size and regional distribution. As documented in Table 1, the list of existing CFREF recipients range from Laurentian University (9,500 students enrolled in 2019) to the University of Toronto (just over 91,000 students enrolled in 2019). In fact, CFREF primarily targets a locus of excellence within an institution, rather than the institution itself.

*Success rates represent the proportion of complete grant proposals that were successful.

Source: CFREF application data. Classification of institutions’ size is based on the Canada Research Chairs (2017).

In sum, by its very nature, CFREF builds on other programs that allow for research inquiries to be pursued and for leadership in targeted research areas to emerge in the first place. For instance, eight Canada Excellence Research Chairs (CERC)^{Footnote 11} and Canada C150 Research Chairs award holders (current and emeritus) and up to 83 Canada Research Chairs (CRC) award holders have been involved in CFREF grants awarded in the first competition. During interviews conducted as part of the evaluation, representatives from funded institutions also emphasized the contribution of CFI grants in building the research environment that positioned the institution to successfully compete for a CFREF grant. Put simply, in cases where a broad eco-system of research funding and infrastructure have already enabled institutions to establish programs of research excellence, CFREF provides a springboard for institutions to further expand their institutional vision and leadership role.

CFREF’s niche in relation to NFRF and the Superclusters

When assessing the niche of CFREF in the overall ecosystem of research funding in Canada, two other programs, the New Frontiers in Research Fund (NFRF) and Innovation Superclusters Initiative, were flagged by stakeholders as being of particular interest in terms of whether there would be connections or synergies with CFREF.

New Frontiers in Research Fund

When planning was initiated for this evaluation, NFRF was a very new program. It was launched in 2018-19, with an initial investment of $275 million over five years, and $65 million annually thereafter (Government of Canada, 2020d). NFRF provides funding through three streams to support ground-breaking research in Canada. Of these, NFRF’s Transformation stream is perhaps the most comparable to CFREF as it aims “to support large-scale, Canadian-led interdisciplinary research projects that address a major challenge with the potential to realize real and lasting change” (Government of Canada, 2020c).

While the Transformation stream of NFRF and CFREF share the broad goal of supporting large-scale, world-leading Canadian research, they also have several distinct characteristics. Most notably, NFRF differs from CFREF as it does not provide an institutional grant, rather supports a team of researchers with one Nominated Principal Investigator, who becomes the award holder. Although CFREF grantees may establish collaborations with international and/or inter-institutional players, the institutional focus of CFREF uniquely positions it as a vehicle for drawing together faculty and HQP from different disciplines within an institution and building internal strengths in support of a common institutional vision. CFREF is also intended to support institutional programs of research in specific priority areas, while NFRF may fund projects outside of these research areas.

Given these key differences, the evaluation determined that CFREF and NFRF fill unique roles in the Canadian funding landscape, however, there may be potential for synergies between the two programs. In particular, the scale of funding that NFRF provides, through its Transformation stream, combined with its objective to support to large-scale, interdisciplinary research projects could potentially support research in some of the areas funded by CFREF. During interviews, some stakeholders spoke about ways in which they might extend their research activities and maintain momentum once their CFREF grants end. Although it is too early at the time of this evaluation to comment on the nature of any subsequent funding or speculate about grantee’s participation in future funding competitions, several grantees expressed a view that NFRF might serve as a potential “next step” to provide some support to sustain their research activities following the end of their CFREF grants. These respondents felt that NFRF might be a particularly good fit for some of the projects currently supported by their CFREF grants especially given the interdisciplinary research teams and multidisciplinary approaches to research that have been developed under CFREF to date.

Superclusters

At the time of the evaluation, there was an interest in better understanding how CFREF and the Innovation Superclusters Initiative would interact. Unveiled as part of the 2017 budget and managed directly by Innovation, Science and Economic Development Canada (ISED), this initiative operates with a total budget of up to $950 million over five years, starting in 2017-18 (Government of Canada, 2020b). To date, five superclusters have received funding to address challenges concerning digital technology, protein industries, manufacturing, artificial intelligence and oceans. These superclusters are led by industry consortia, and the private sector is expected to match dollar for dollar the federal investment. Each supercluster involves a large number of industrial partners, as well as some public and academic entities.

The primary purpose of superclusters is to “energize the economy and become engines of growth” (Government of Canada, 2018e). As such, the initiative shares with CFREF a common goal of sustaining innovative economic growth in Canada. Both programs also aim to recruit and retain new talent and promote multisectoral collaborations and partnerships, which help position Canada as a world-leading innovation hub. Beyond this, however, the two programs operate in largely different spheres. As noted during interviews, the fact that CFREF is decisively led by academia and the superclusters are just as decisively led by industry shapes their actions and priorities. Whereas CFREF is pursuing innovative research programs with promising scientific, research or social applications, the superclusters are focusing on the later stages of the R&D continuum, with the expectation that short-term commercialization benefits will be realized through new products, new manufacturing processes, new technologies or new commercial strategies.

In the larger picture, the two initiatives play different roles and the extent to which researchers and/or projects under these initiatives will formally or informally interact remains uncertain. It is worth noting that some of the universities involved in CFREF grants do participate in the activities of superclusters, however, there have been no instances yet where a CFREF research grant and a supercluster directly collaborate based on a shared strategy.

2.2 Alignment with current government priorities

The federal context has evolved considerably since CFREF was launched in 2014. In particular, the change in government resulting from the 2015 federal election led to a new approach in setting priorities related to research and innovation.

Priorities as of 2014

As already noted in the description of CFREF, the first two competitions under CFREF required applicants to address at least one of the priority areas included in the 2014 Science, Technology, and Innovation Strategy. The evaluation confirms that all 18 grants are directly reflecting these priorities. As indicated in Figure 1, these grants are almost equally distributed among priority areas, except for advanced manufacturing where only one grant focussed primarily on it (quantum materials). However, these priority areas are not exclusive of each other, and most grants also address other priority areas.

Source: TIPS administrative data

Priorities as of 2020

Since the 2015 election, the government has repeatedly emphasized the importance of fostering innovation, supporting scientific research and attracting new research talent (Government of Canada, 2016a, p. 110, 2017a, p. 17, 2018a, p. 85, 2019, p. 121). This has included, among other things, new investments in scientific research, including the NFRF, and the launch of the Innovation and Skills Plan, which contains many components, such as the Innovation Superclusters Initiative and the Strategic Innovation Fund (Government of Canada, 2020a)(Government of Canada, 2019). In addition, the government has pursued horizontal priorities, particularly as they relate to EDI and support to ECRs. In announcing, in its 2018 budget, an investment of nearly $4 billion in Canada’s research system, the federal government underscored that this investment “is tied to clear objectives and conditions so that Canada’s next generation of researchers―including student, trainees and early career researchersis larger, more diverse and better supported” (Government of Canada, 2018a, p. 82).

The priority areas included in the 2014 Science, Technology, and Innovation Strategy, while still relevant, are no longer acting as the primary driver to inform funding programs across government. There is scope within the program framework which allows for government to review these areas before new competitions. This is reflected in the recent practice of funding agencies to identify priority areas in the context of particular research funding program competitions. For instance, the 2016 CERC competition used the priority areas of the 2014 Science, Technology, and Innovation Strategy, but also added new research areas related to social inclusion and innovative society, and open areas of inquiry to be of benefit to Canada (Government of Canada, 2018d).

In addition to the priority areas that were targeted at the time of the CFREF competitions, the evaluation confirms that the CFREF program continues to support the broader horizontal federal priorities, such as those relating to EDI and ECRs. In terms of EDI, at the outset, applicants to the program had to include an “equity plan outlining how career and training benefits derived from the opportunities associated with the initiative will be made available to individuals from the fours designated groups (women, visible minorities, Indigenous Peoples and persons with disabilities)” (Government of Canada, 2016e). During interviews, representatives from funded institutions were very conscious of the importance placed on EDI and on the expectation that progress in that regard would be achieved and properly documented.

What has not been as clearly defined in the program information are CFREF’s objectives and expectations of grantees as they relate to supporting ECRs. As noted earlier, supporting ECRs was introduced as a government priority in Budget 2018, two years after the last CFREF competition in 2016. CFREF grants are logically intended to enhance the environment in which participating ECRs evolve. However, at a more fundamental level, the evaluation found that the expected impact of CFREF on ECRs remains to be more clearly defined, regardless of whether or not senior management determines that the program has a role to play in implementing this priority. TIPS currently collects information on the number of ECRs involved in the initiatives at mid-term which suggests that the program’s contribution to this priority is of interest to management.

3.0 The implementation of CFREF grants

This section of the report turns to the actual implementation of CFREF grants at the institutional level. The evaluation first relied on the mid-term reviews administered by TIPS to determine whether or not the implementation of the grants from the first competition has met expectations. Specifically, the TIPS Steering Committee approved all five grantees from the first cohort for continued funding with recommendations. The interviews, case studies and administrative data review conducted as part of the evaluation complemented this exercise by providing a high-level overview of the collective experience of the 17 grantees to better understand the range of implementation strategies used to date, challenges encountered and lessons learned.

3.1 Grant governance models and funding allocation processes used

Governance structures

The program guidelines provided considerable flexibility for grantees in establishing their governance structure. As part of their proposals, applicants were asked to describe the approach they were planning to use to ensure that the grant would be successfully implemented, including the proposed governance structure and the accountability and decision-making processes (Government of Canada, 2016b).

While no governance structure is identical, and ongoing adjustments are implemented as required, the grantees have typically included at least the following components:

• An overarching committee (e.g., a steering or executive committee) that oversees the institution’s progress in implementing the grant and in ensuring that the vision behind the scientific and institutional strategies being implemented through CFREF is adequately integrated in the institution’s strategic direction. The committee normally includes the vice-president of research (or equivalent), along with other senior administrators (e.g., deans) and faculty members. The grant leads (scientific and/or administrative), as well as other committees, report to this committee;
• Advisory committees, involving both internal and external scientific experts and partners, that provide advice or make recommendations in all aspect of the grant management, including the allocation of funds, as applicable;
• A peer-review committee that involves external experts, supporting the selection of specific projects to be funded through the grant; and
• An executive director position (or equivalent) who manages activities and processes on an ongoing basis. Additional team members are normally assigned to support the executive director, including, for instance, communication officers and administrative support (who may be assigned on a part-time or full-time basis).

In some cases, the CFREF grant has been integrated into existing institutes such as the Institut Quantique in Sherbrooke, the Stewart Blusson Quantum Matter Institute in Vancouver, the Sentinelle Nord in Laval and the Global Institute for Food Security in Saskatchewan. In other cases, such as Medicine by Design in Toronto, the grant has been set up as a horizontal initiative that is managed with the involvement of applicable faculties/departments. In all cases, however, the grantees have created a distinct brand for their CFREF grant, including a website and other communication means (e.g., presence on social media).

Key features that support the strategic direction of the grant

The experience gained by the first 17 CFREF grantees is shedding light on factors that contribute to a governance structure that can effectively support the strategic direction of the grant.

First, the flexibility that CFREF provides to each grantee in determining the most appropriate governance structure was perceived as a key strength of the program by many interviewees. Each postsecondary institution in Canada brings its own history, distinguishing features, organizational culture and governance model. The type of scientific research being undertaken through each grant and their unique strategic direction further adds to the variety of governance approaches that may be required.

Second, having the vice-president of research (or equivalent) directly involved in the management of the grant is perceived as particularly beneficial. As the fundamental goal of CFREF is to allow for the implementation of institutional and scientific strategies that will strengthen the positioning of the institution in a specific field of research, it is desirable to involve individuals who can effectively bridge the CFREF grant and the entire strategic vision of the institution.

Third, the range of activities undertaken, and the various monitoring and reporting requirements associated with the grant, necessitate considerable time and resources. Having an administrative team specifically dedicated to the grant, including a full-time executive director (or equivalent), is emerging as a good practice. As noted during interviews, dedicating personnel, including an administrative lead, to manage the administrative aspects of the grant is critical for ensuring that the scientific lead has sufficient time to dedicate to managing the scientific aspects of the grant.

Finally, the ability of grantees to connect and share lessons learned offers many benefits, and this appears particularly true when it comes to sharing lessons learned on the effective, strategic management of a large institutional grant such as CFREF (e.g., at the annual summit for CFREF grantees or through individual exchanges).

Challenges encountered

Some of the funded recipients identified challenges experienced during the implementation of their grant, particularly as it relates to the early period of implementation. The evaluation primarily drew on the case studies of Competition 1 awardees to identify these challenges, although the experiences of Competition 2 awardees were also considered. The three primary challenges that emerged are:

• The time and resources required to establish the governance structures and the detailed implementation process for the grant: For grantees, a considerable portion of the first year of the grant implementation was devoted to establishing the required governance structure, including the recruitment of the core team tasked with managing the grant. In doing so, grantees often had to modify the governance structure from what they had initially included in their application to address the full range of practical considerations that the implementation of their grant entailed. To various extents, this process deferred the actual implementation of the scientific strategy and required some grantees to allocate more resources than initially anticipated to the administration of their grants. This challenge was more predominant among Competition 1 grantees, which were provided with a particularly short timeframe to develop their proposals.^{Footnote 13}
• Uncertainties related to the range of potential governance models: As part of the application process, CFREF provided applicants with considerable flexibility in determining the governance structure that they would use to manage the overall grant, including the allocation of funding. While grantees highlighted several advantages associated with this flexible approach, some noted that it was sometimes challenging to understand whether the structure they had adopted would meet any set expectations on the part of the program. For instance, during interviews, some grantees indicated that they would have appreciated additional information to clarify the instructions posted on the program’s website but were not able to secure it. Without being more prescriptive, it appears that additional assistance from TIPS in guiding some grantees as they finalize their governance model would be helpful, particularly in light of the size of the grants that CFREF provides.
• Managing researchers’ expectations during the implementation of the scientific strategy: Following the awarding of grants, institutions have had to implement processes that would allow for the allocation of funds to specifically support the goals and objectives of the scientific strategy initially proposed in their application. During interviews, grant administrators have emphasized the importance of clearly communicating this vision to all potential researchers who may wish to undertake research with the support of the CFREF grant. In other words, the very notion that only research activities directly aligned with the scientific strategy would receive funding has, at time, been challenging to communicate to some researchers who wished to pursue their own research agenda. In this context, grant administrators have had to manage expectations, present the various steps initially required to be able to allocate funding, under which conditions, and the timeframe that this would entail. For some grantees, this created some tensions between the grant administrators and the researchers. According to some key informants, this is something that any new grantees should plan for.

Funding allocation processes

CFREF expenditures

In accordance with CFREF administrative guidelines, institutions have been allocating their grant funding to cover both the direct and indirect costs of their scientific and institutional strategies. Notably, indirect expenses were limited to a maximum of 25% of the total CFREF grant. The range of funded activities reflects what would be expected of any research project or program, and typically covers:

• the recruitment of new faculty members and other research professionals;
• scholarships or other forms of support to students and postdoctoral fellows; and
• the provision of support to research activities (seed funding, start-up package, equipment and supplies, travel, etc.).

The 17 grantees’ total projected expenditure over the grant term, as well as the actual expenditures to date (as of 2018-19), is shown in Table 2 below.

* Amounts in this column represent the sum of actual spending as of 2018-19 and the projected spending for the remaining of the seven years of the grant term.

Source: 2018-19 annual financial reports

The various activities that are funded by the CFREF grant may also be partially financed by other sources. This is particularly true when it comes to salaries and other related expenses for new faculty members or research professionals and indirect costs. Considering more specifically the experience of Competition 1 grantees in allocating funding for compensation, as shown in Table 3, only 3% of compensation-related expenses used as of 2018-19 were for new faculty appointments, which indicates that CFREFs have been successful in financing new faculty appointments from sources other than the CFREF grant.

Table 3 shows that a total of $13,040,187 has been spent by Competition 1 grantees on compensation-related expenses for research administrative support as of 2018-19. This represents 23% of the total amount spent on compensation-related expenses by Competition 1 grantees as of 2018-19, and 11% of the total grant amount spent by Competition 1 grantees as of 2018-19 (as outlined later in the report, in section 6.1).

Source: 2018-19 mid-term reports

To ensure that the salary of new faculty is paid from a sustainable source, CFREF grants cannot be used to pay for the salaries of faculty who also receive other tri-agency funding. Instead, funded institutions have opted for alternative strategies to support faculty. Some grantees have used CFREF funds to bridge individuals (e.g., covering the first year of salary only) or to cover a portion of the costs, while other institutions have opted for not using CFREF funds to cover salaries, rather using it to provide other support such as start-up packages, access to research personnel or research funds. (Note that grantees’ experiences with the salary restriction are further discussed in section 4). These findings were also confirmed by faculty responding to the survey. As illustrated in Figure 2, over 70% of faculty who had been involved in CFREF indicated that they had received support for their research activities and to hire students and/or staff. No more than 22% of the respondents indicated that their salary as a faculty member had been paid, in part at least, through the CFREF grant. Other forms of support included funding to attend conferences, workshops and other professional activities, as well as assistance with research facilities.

Source: Survey of CFREF grant team members

As shown in Table 2, 20.2% of CFREF funds that have been spent across the 18 grants as of the 2018-2019 were for indirect costs. This percentage is slightly higher than what grantees initially expected to spend on indirect costs over the duration of the grant term (17.4%). The amounts spent may not reflect the actual total of indirect cost associated with delivering these grants, given that grantees could use resources leveraged from the institutions to support implementation.

As CFREF grants are large and administratively complex to implement, institutions awarded as part of Competition 1 had spent a third of their grants at mid-term on average (33%), which is lower than what they had originally projected immediately following the award of the grant (i.e., 50%). The challenges grantees had experienced in quickly ramping-up the grant were also evident among Competition 2 grantees. Their spending followed a similar pattern, with 19% spent in 2018-19 despite the originally projected 38% by that time. This provides some context for why some grantees have requested and received approval for no-cost grant term extensions of two years as part of the mid-term review to be able to use all grant funds they have been allocated. The use of funds over time should be carefully monitored going forward, particularly given that the current COVID-19 pandemic may cause additional delays.

Allocation processes

Funded institutions have been using a mix of existing and new processes to allocate the funding in accordance with their scientific and institutional strategies. With respect to the allocation of funding for recruitment of researchers and HQP, some institutions have relied on existing policies and processes to guide their funding allocation. In other cases, such as seed funding and other forms of support to research projects, institutions have developed new processes and established new committees to ensure the involvement of the required subject area expertise. This was particularly the case for competitive processes involving peer-review committees.

Researchers who received financial support had most commonly accessed financial support for their research activities through a competitive process (68%) as opposed to an allocation process (24%) (n=357). It was relatively rare to have accessed funding in both ways at the time the survey was administered (3%). Directly allocating funds to researchers generally occurred as part of the start-up phases of the grants to quickly distribute initial research funding, or as part of grantees offering start-up packages to new hires.

CFREF grantees are expected to uphold the principles of fairness, rigour and transparency in allocating funding. However, these processes tend to be complex and involve a degree of discretion and latitude. As part of the survey, 74% of faculty who had been successful in receiving funding through a competitive process indicated that they had no concerns with the process their institution had used (see Figure 3). Given that this group of respondents were successful, it would have been reasonable to expect this percentage to be higher. The most common concerns were lack of transparency, unclear evaluation criteria and perceived conflict of interest in decision-making in terms of funding allocation. In a couple of instances, CFREF leads acknowledged that there were times when communication could have been improved or changes to a process had caused some confusion. Part of this was due to the learning curve associated with getting the process set up.

Source: Survey of CFREF grant team members

Even though some concerns were identified, during both the survey and the interviews, the evaluation found that the overall strategies in place to allocate funding reflect the current parameters established in CFREF guidelines. The funding mechanisms also appear to be broadly aligned with the scientific and institutional strategies even though some grantees were currently in the process to narrow their focus further in terms of the number of projects funded or priority subsubject areas (as discussed earlier).

3.2 Leveraging of additional funding

Contributions from funded institutions and their partners

In their grant proposals, applicants were asked to demonstrate the willingness of the lead institution to commit internal resources to support the proposed CFREF initiative, as well as the ability of the institution to leverage additional resources and promote knowledge mobilization through partnerships.

Although CFREF does not have specific requirements to leverage matching funds, the ability of funded institutions to secure funding from partners^{Footnote 14} to support their scientific and institutional strategies is one of the the subcriteria considered during the selection process (Government of Canada, 2016b, p. 8). Applicants are asked to provide details on past success in leveraging funds for the institution as a whole and in the specific research area(s) targeted by the CFREF, in addition to details on prospective leveraging plans by identifying key partners who have expressed an interest in collaborating, as well as federal (e.g., tri-agency and CFI) and non-federal funding sources and programs that will be accessed for operating funds to build and strengthen the initiative.

Grantees have successfully leveraged a significant level of funding in support of their strategies. At the end of 2018-19, $1.3 billion had been committed by the funded institutions and their partners for the seven-year period covered by the grants. Considering that the program awarded a total of $1.25 billion in grants, this means that the CFREF funding could end-up being matched or surpassed by the leveraged funding. Nonetheless, it is important to note that in some cases these reported leveraged funds are not exclusively used by researchers involved in the CFREF grants. That is, the reported amounts of leveraged contributions include funds contributed to other beneficiaries who are not involved in the CFREF grants. To be more specific―and as illustrated in Figure 4―by the end of the fiscal year 2018-19, a total of $367.4 million has been committed by Competition 1 lead institutions and their partners. For Competition 2, $965.3 million had been committed from the same sources. This support includes predominantly cash contributions, supplemented by in-kind contributions. In total, 85% of the contributions committed by funded institutions and 72% of the contributions committed by partners are in cash.

Source: 2018-19 annual financial reports submitted by funded institutions

Looking more closely at the contributions from partners (competitions 1 and 2 combined), 43% of these contributions came from the public sector, while 26% came from the private sector and 21% came from academic institutions (excluding the funded institution). Of note, 20% of contributions from partners came from entities located outside of Canada (see Table 4).

*Excluding contributions made by the host institutions.

Source: 2018-19 annual financial reports submitted by funded institutions

At the time of this report, not all contributions committed by the funding institutions and their partners had been secured. As expected, with one additional year of implementation done, grantees from Competition 1 had secured a greater portion of their committed contributions. As illustrated in Figure 5, 62% of the contributions from the funded institutions and 67% of the contributions from the partners had been secured as of the end of 2018-19 fiscal year.

Source: 2018-19 annual financial reports submitted by funded institutions

As for Competition 2, 31% of the contributions from the funded institutions and 49% of the contributions from partners had been secured as the end of 2018-19 fiscal year.

Source: 2018-19 annual financial reports submitted by funded institutions

Additional funding from the granting agencies and other federal departments^{Footnote 15}

As part of the mid-term review process, Competition 1 institutions reported on the funding they have received from the granting agencies (other than through CFREF), which has contributed to advancing their respective scientific strategies. A total of $194 million has been awarded by these agencies since drafting the CFREF proposals^{Footnote 16}. As illustrated in Figure 7, close to half of this funding has come from CIHR, while NSERC has been the other main contributor. This suggests that there are synergies between CFREFs and other agency and tri-agency funding programs (this conclusion was discussed in section 2.0 of this report).

Source: Mid-term reports submitted by Competition 1 grantees

In addition, Competition 1 grantees have reported cash and in-kind contributions worth $45 million (as of 2018-19) from a wide variety of other federal organizations and agencies including, but not limited to, Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, Natural Resources Canada, the Canadian Space Agency and the National Research Council of Canada.

Support being sought for sustaining the transformative changes of the CFREF grants

The CFREF logic model expresses an expectation that CFREF grantees are to sustain at least parts of the transformative changes of the CFREF at the institutional level. Specifically, the expected legacy of the CFREF program are articulated in the long-term outcomes that are set to occur after the seven-year grant has ended (i.e., international recognition, benefits from research in priority areas and economic advantages for Canada). In their mid-term reports, grantees are asked to explain how the momentum and transformative changes will be sustained for the remainder of the grant’s term and beyond.

How grantees addressed this varied, but their explanations largely focused on what other funding would be sought to sustain parts of their initiative and how they envisioned to position the initiative or institution in certain areas (e.g., partnerships, interdisciplinary research, commercialization), mirroring some of the medium- and long-term outcomes of the CFREF program.

Plans were also briefly discussed in interviews with all CFREF leads and VPs of research at funded institutions. Some expected to sustain at least parts of the research activities fostered by CFREF through continued institutional support (in terms of both direct support such as operating funds, as well as indirect forms of support through fostering interdisciplinary and cross-departmental collaborations); accessing traditional, more discipline-specific funding opportunities (e.g., basic research programs provided by the three granting agencies), as well as benefiting from their increased capacity for interdisciplinary research to access more provincial, national and international funding mechanisms (e.g., NFRF).

Given that it is still early in the program’s lifecycle, much of the funding mentioned had not yet been secured, it is not surprising that the overall results from the mid-term review highlighted sustainability of the CFREFs as a concern and suggested that grantees should be asked to continuously update their plans describing how they will sustain the transformational changes to minimize end-of-grant impacts. Finally, it is important to note that the program literature does not provide detailed expectations of grantees in this regard, in part due to the unique scientific and institutional strategy of each initiative.

4.0 CFREF participants

As funded institutions implement their scientific and institutional strategies, they are expected to attract and retain a pool of high-caliber, diverse and interdisciplinary researchers. As the CFREF program is expected to build on existing institutional strengths, it is important to note that the program is expected to both retain those who were already at the institution when the CFREF grant was received, as well as to contribute to retaining participants involved in the longer term. The program literature does not currently suggest what proportion of participants is expected to be retained. Further, it does not place a relative importance on retention of existing personnel and trainees vs. recruitment of new.

Participants are expected to operate in an enhanced training environment that provides a range of research and training opportunities. The evaluation focused on better understanding the range of participants who have been engaged through grant activities carried out during the evaluation period and the nature of the training and research environment the CFREFs contributed to. The extent to which CFREF grants will ultimately increase the capacity of funded institutions to retain these individuals in the longer term is an issue that will be revisited later in the grants’ lifecycle.

A CFREF participant is defined as an individual who is either involved in conducting research related to a CFREF grant or is involved in the administration and management of the activities carried out as part of the grant. The list of participants may therefore include faculty, research trainees (undergraduate, graduate, doctorate), HQP such as postdoctoral fellows, research technicians, research associates, other technical or research personnel, and administrative staff at both the lead and partner institutions (TIPS, 2017a).^{Footnote 17} Participants may or may not have received funding directly from the CFREF grant. Unless otherwise noted, statistics on the involvement of participants include both current and past participants.

4.1 Overview of participant pool

As of March 2019, CFREF grants had engaged a total of 6,780 participants. The five institutions funded as part of Competition 1 have involved 3,294 participants, whereas the 13 institutions funded as part of Competition 2 (with one less year of grant implementation) have involved 3,486 participants).

There was a significant variation in the number of participants that each grant has engaged. Keeping this in mind, on average, each grant has involved 87 faculty, 49 postdoctoral fellows, 137 graduate students and 35 undergraduate students. Out of the 431 faculty who participated in Competition 1 grants, 83% had received funding from the grant. Table 5 provides further information on the category of participants.

Source: Mid-term report for Competition 1 and the sample for the survey conducted by evaluation on CFREF participants for Competition 2. This data reflects the number of CFREF participants since the start of the CFREF grants (2015 for Competition 1 and 2016 for Competition 2) up until 2019. (The inclusion criteria and the extent of involvement in the CFREF grant for participants in the sample may have differed among the grantees).

Faculty members

The majority of faculty members who have participated in activities supported through Competition 1 grants were full professors (see Figure 8). Notably, of total faculty currently involved in CFREF grants, 50 were Tier 1 Canada Research Chairs (CRCs), 33 were Tier 2 CRCs, and eight were holders of Canada Excellence Research Chairs. Moreover, of the total number of faculty involved, 18% were identified as ECRs in the mid-term reports. Among faculty survey respondents who reported to have come from another Canadian or foreign institution, a large majority (88%; 37 out of 47) identified as ECRs.^{Footnote 18}

Source: Mid-term reports submitted by Competition 1 grantees

Although most faculty members were already affiliated with their institutions, data from the mid-term review reports indicate that Competition 1 grantees did proceed with recruitment activities. At the time of the evaluation, they had recruited 57 new faculty members, including 27 individuals recruited from outside Canada. An additional 22 members were recruited from the funded institutions themselves, while seven were recruited from other institutions in Canada (see Table 6). The high number of faculty recruited from within the home institution may reflect that it is easier to recruit from within the institution during the first years of grant implementation.

Source: Mid-term reports (2018-2019)

One challenge that has been faced in recruiting new faculty members is the salary eligibility restriction included in the CFREF guidelines (as explained in Section 3.1). Put simply, any faculty or researcher whose salary is paid out of a CFREF grant cannot receive other support from the granting agencies (e.g., Discovery grants) (Government of Canada, 2018c). The rationale for including this restriction is to ensure that the source of funding for new hires is sustainable. In the long term, universities are expected to be paying the salaries of faculty from other sources than the CFREF grant. The evaluation confirms that this restriction has created difficulties particularly for the smaller institutions because providing new faculty with salary support using CFREF funds would come at the cost of them being able to access to other tri-agency funding. Nonetheless, it appears that recipients have learned how to work around these challenges over time by finding alternative means to attract and support faculty (e.g., leveraging support of lead and partner institutions to cover salary support for recruited faculty while using CFREF funds to offer additional incentives such as start-up packages for new faculty to establish their research labs and hire research and administrative personnel).

The nature of faculty appointments varied, including new tenure-track positions that were created at the lead institutions (permanent and time-limited). Others were affiliate or adjunct faculty appointments with partner institutes or affiliated institutions. During interviews, key informants noted that not all appointments were entirely attributed to CFREF, but the scientific strategy pursued by the lead institution had a direct influence on the expertise sought from new appointees in several cases.

Highly qualified personnel (HQP)

Over 2,800 HQP have had the opportunity to engage in activities carried out as part of Competition 1 grants, and this number is projected to increase to over 3,400 by the end of the grant terms. As indicated in Table 7, Competition 1 grantees have recruited^{Footnote 19} HQP from their institution, as well as from other Canadian and foreign institutions.

Source: Mid-term reports (2018-2019)

4.2 Equity, diversity and inclusion (EDI) consideration

As part of their proposal for a CFREF grant, institutions were required to include an EDI plan to describe their efforts to ensure that the career and training benefits derived from the activities undertaken as part of the grant would be accessible to individuals from the four designated groups: women, visible minorities, Indigenous Peoples and persons with disabilities (Government of Canada, 2016b). The inclusion of an EDI plan was a requirement for applicants so as to demonstrate the strength of their research culture and their commitment to ensure equity within the program. The grantees’ EDI plans are monitored as part of the annual progress reports and reviewed as part of the mid-term review of the grants.

Evaluation findings from case studies and key informant interviews indicate that many funded institutions had already (i.e., pre-CFREF) worked to establish institution-wide strategies to identify and address systemic barriers that limit the participation of the four designated groups in their academic and research programs and activities. According to many interviewees, these institution-wide EDI strategies provided a foundation for the EDI plans that were developed as part of the CFREF proposal. In many cases, key informants noted that these institutional strategies have been integrated in the ongoing management of their CFREF grants and other, more grant-specific, EDI strategies have been adopted where possible (i.e., efforts to recruit HQP from the four designated groups within smaller research teams/projects).

Despite these efforts, some key informants recognized that there is more work to be done to increase the diversity and participation of these groups. Findings show that there is some underrepresentation within the CFREF teams and their governance structures. This was highlighted both in the overall results from the mid-term review and administrative data examined as part of the evaluation. Some of the challenges identified by key informants include the perception that, for certain disciplines, there is a more limited pool of researchers from the four designated groups, as well as challenges in recruiting candidates to the various geographical locations of the lead institutions. In some universities, the requirement for participants to engage in French-language activities (e.g., teaching courses in French), can also pose a barrier to recruitment within the teams, as this further limits the pool of potential candidates who can be recruited. Some key informants also acknowledged that the current governance structures of the CFREF grants lack adequate representation of individuals from the four designated groups.

As shown in Figure 9, the representation of individuals from the four designated groups among CFREF participants reflect to some extent the diversity of Canada’s overall population; however, gaps persist for some of the four designated groups more than others. The comparison of representation data of the CFREF participants gathered using the program’s self-identification questionnaire to that of Canada’s overall population and labour market availability (Statistics Canada, 2016 Census), indicates that further efforts are required to achieve equitable participation within the program, particularly as it relates to Indigenous Peoples (0.5% of CFREF participants vs. 4.9% of Canada’s population and 4.0% of Canada’s labour market), persons with disabilities (1.8% of CFREF participants vs. 14.0% of Canada’s population and 9.1% of Canada’s labour market) and women (34.7% of CFREF participants vs. 50.9% of Canada’s population and 48.2% of Canada’s labour market).

Source: CFREF self-identification data (n=4,392), 2016 Census (Statistics Canada) and 2006 Employment Equity Report (Human Resources and Skills Development Canada)

As illustrated in Figure 10, individuals who identify as women represent almost a quarter of CFREF faculty, which is below the percentage of individuals who identify as women among full-time university faculty^{Footnote 20} within Canadian universities (40.6%; University and College Academic Staff Survey [UCASS], 2018-2019). Similarly, the representation of individuals who identify as Indigenous Peoples among CFREF faculty (0.4%) is relatively low compared to their representation among university faculty and researchers^{Footnote 21} in Canada (2.0%). To a lesser extent, individuals who identify as a visible minority among CFREF faculty (16.6%) were proportionally lower than among university faculty and researchers in Canada (19.4%).

Source: CFREF self-identification data (n=4,392), Full-time University and College Academic Staff System (UCASS, Statistics Canada, 2018-2019); and Survey of Postsecondary Faculty and Researchers (SPFR, Statistics Canada, 2019)

The literature review suggests that the extent of underrepresentation of women among full-time faculty at Canadian universities is relatively higher in the STEM-related fields (Canadian Association of University Teachers, 2016). Given that the 18 CFREF grants are largely associated with natural sciences and engineering, the underrepresentation of women among full-time faculty at Canadian universities in these fields may be one contributing factor towards the magnitude of underrepresentation of women among CFREF faculty presented in Figure 10. As shown in Figure 11, the representation of women among full-time faculty at Canadian universities is relatively lower in the subject areas of agriculture, natural resources and conservation (31.0%); physical and life sciences (27.8%); and architecture, engineering and related technologies (17.4%). Nonetheless, as noted above, it has been recognized that more work needs to be done in order to increase the participation of women and other underrepresented groups. Data on the representation of individuals identifying as members of visible minorities, Indigenous Peoples or persons with disabilities among full-time professors in Canadian universities was not available by subject area, thus, discipline-specific comparisons were not included.

Source: Full-time University and College Academic Staff System (FT-UCASS), 2018

The representation of individuals from the four designated groups was also examined in terms of CFREF HQP^{Footnote 22} and compared to relevant parameters in the Canadian postsecondary sector. Self-identification data on the representation of individuals from the four designated groups among CFREF participants indicate greater diversity to some extent among HQP compared to faculty. Specifically, CFREF participants self-identifying as women (36.6%) or as members of a visible minority (27.3%) had a higher representation among HQP compared to faculty (24.1% and 16.6%, respectively). In contrast, the proportion of individuals who identify as Indigenous Peoples is comparable between faculty and HQP (0.4% for both groups). Similarly, the proportion of individuals who identify as persons with disabilities is comparable between faculty (1.3%) and HQP (2%).

As illustrated by Figure 12, the proportion of CFREF HQP who identify as women (36.6%) is comparable to that of holders of doctoral degrees in Canada (39.1%). Similarly, the proportion of HQP who identify as members of a visible minority (27.3%) is comparable to that of holders of doctoral degrees in Canada (31.1%). In addition, the proportion of CFREF HQP who identify as Indigenous Peoples (0.4%) is less than half of the same proportion of holders of doctoral degrees in Canada (1.0%). Finally, the proportion of CFREF HQP who identify as persons with disabilities (2.1%) is half of that same proportion of holders of doctoral degrees in Canada (4.0%).

Source: CFREF self-identification data (n=4,392); and Postsecondary Student Information System (PSIS, Statistics Canada, Centre for Education Statistics, 2016-2017)

4.3 Training environment

The various lines of evidence employed in the evaluation provided insight into how the CFREF grant has enhanced the training environment at lead institutions. In addition to providing additional funding for hiring and training of HQP, the CFREF grant allowed lead institutions to enhance their training capacity through leveraging the support of partners (e.g., funding and expertise).

In particular, CFREF has been shown to increase the extent of interdisciplinary training that is available to participants through coordinated, interdisciplinary activities occurring among the various departments involved in the grant within the lead institution. This enhanced coordination facilitated activities such as the increased co-supervision of students by faculty members from different departments, the establishment of interdepartmental research labs and shared use of research facilities, as well as fostering more frequent interactions between trainees from different research areas (e.g., large meetings involving multiple research labs).

Moreover, across all five case studies conducted as part of this evaluation, the transdisciplinary training environment fostered by the CFREF grant was mentioned as an attraction for faculty and HQP alike. Specifically, the involvement of HQP in interdisciplinary research teams was perceived as a unique feature of CFREF teams that enhances their employability and career-development. The interdisciplinary research environment that was facilitated by CFREF was also praised by faculty members, including ECRs.

According to multiple groups of key informants, the CFREF grants provided an opportunity to complement pre-existing training programs at lead institutions. That is, CFREF has supported the establishment of training and career-development opportunities that offer participants enhanced benefits compared to the traditional training opportunities that were previously available, including a wide range of non-academic skills such as commercialization and translation skills, as well as communication skills. Moreover, the additional funding provided by CFREF has supported enhanced exposure of trainees to national and international networks of researchers through exchange programs, internships and hosting high-quality visiting researchers. Notably, some of the grantees indicated that funding and expertise leveraged from partnerships will be drawn upon to help sustain the enhanced training environment, which has been supported by CFREF, following the end of the grant term.

CFREF was also reported to enhance the training environment at lead institutions through offering participants increased accessibility to state-of-the-art research facilities and equipment. CFREF has provided additional funding for the purchase and maintenance of equipment that is essential for research and training purposes. In addition, the CFREF grant has supported lead institutions in leveraging additional support from partner institutions (e.g., shared use of facilities) as well as additional funding for the establishment of research facilities.

Both faculty and HQP highlighted several opportunities that were made available with the support of the CFREF grant. Most commonly, as a result of their participation in the CFREF grants, survey respondents highlighted that they have had the opportunity to engage with high-caliber researchers (69%), participate in projects leading to the creation of new knowledge (69%) as well as the extension/application of knowledge (68%), to participate in multidisciplinary research collaborations (60%) and high-risk, high reward research projects (49%) to a great or very great extent. HQP, in particular, also noted that their participation in CFREF funded initiatives has enhanced their skills and expertise in undertaking research, preparing reports and publications, working in groups, communicating and interacting with others, and managing projects. It is important to note that the development of the training environment is considered still in early stages given the short period since the launch of the CFREF program.^{Footnote 24} Nonetheless, the administrative data review and key informant interviews provided various examples of the training opportunities available to CFREF participants, including a wide range of non-academic skills training such as commercialization and knowledge translation skills (e.g., entrepreneurship, patents and intellectual property rights); communication skills; opportunities for international exchange and internships; as well as site visits to conduct field work in remote locations.

5.0 Partnerships, collaborations and infrastructure

CFREF was expected to facilitate the establishment of new partnerships and collaborations, as well as strengthening those that predate the CFREF grant. It is worth noting that a “partnership” refers to the involvement of an institution or organization (e.g., government, industry, associations, not-for-profit organizations or other institutions) that is expected to contribute in-kind or cash contributions to support the CFREF initiative. A “collaboration” refers instead to the involvement of an individual (from academia or from other sectors) who plays an active role in the research and research-related activities of the CFREF initiative, but is not a CFREF participant (TIPS, 2017a, pp. 5-6). When it comes to partnerships and collaborations in the context of CFREF, the most important aspect is its quality and relevance in relation to the strategic direction of the grant. Further, the evaluation looked at the quantity and reach of partners and collaborators involved in CFREF grants and captured their perceptions of the opportunities that the CFREF grants afforded to their own goals.

The evaluation also captured grantees’ perceptions on the adequacy of their access to infrastructure for implementing their strategies and the contribution of CFI funding to their initiatives.

5.1 Partnerships

At the time of the evaluation, more than 600 partnerships were reported by the 18 CFREF grants (see Table 8). Competition 1 grantees had an average of 54 partners at mid-term, but the range was wide, from 25 to 124 partners.

Source: 2018-19 mid-term reports for Competition 1 and the 2016-18 annual progress reports for Competition 2

As illustrated in Figure 13, the range of partnerships established includes partners from the private sector, other academic and research institutions, the public sector (particularly, provincial governments) and other sectors (e.g., philanthropic organizations). During interviews, particularly those conducted as part of the site visits, representatives from the funded institutions noted that some of these partnerships (such as those with other academic institutions) provide opportunities to collaborate on fundamental research activities, whereas other partnerships, such as those with the private sector, provide opportunities to access specific expertise or technologies or explore issues related to the anticipated commercialization of the technologies. Finally, partnerships with the public sector, such as provincial governments, have allowed research teams to access other forms of research funding support.

Sources: 2018-19 mid-term reports (Competition 1 grantees); and 2016-18 annual progress reports (Competition 2 grantees)

Almost three-quarters (72%) of the total 617 partnerships reported are with entities based in Canada, while the remaining 28% represents international partners.

One quarter (25%, n=67) of the 271 partnerships reported by recipients from the first cohort, were identified as new (i.e., did not predate the CFREF grant), one-half (50%) existed before the grant, and the remaining (25%) were not categorized as either new or existing by the grantees. Of the 67 new partnerships, 49% were with partners in the private sector, 17% were with partners from the public sector, 23% were academic institutions and research organizations, and 8% were partners from other sectors. Of the 67 new partnerships, 54% were international, predominantly with academic institutions and private sector.

Grantees and those interviewed for the case studies reported that many of the pre-existing partnerships had been strengthened following the recipient of the grants as outlined by grantees. Partnerships were strengthened through increases in leveraged funding, exchange of faculty and students as well as shared use of facilities.

Moving forward, some grantees noted that the nature of the partnerships that have been developed with the support of CFREF may change following the end of the grant term. Specifically, the cessation of funding is anticipated to make it challenging to sustain partnerships after the end of the grant term. In contrast, other grantees noted that some of their partnerships may be utilized to sustain components of the CFREF, such as training and support of HQP. Moreover, some grantees note that the enhanced profile and international recognition fostered by the CFREF grant is expected to support the development of partnerships and leveraging of additional funding.

5.2 Collaborations

CFREF grantees have established collaborations with almost 1,500 individuals, with whom they conduct research, publish, present or engage in other knowledge-sharing activities (see Table 9). As for collaborations, the range reported is wide, spanning from 32 to 295 partners among Competition 1 grantees, with an average of 170.

Source: 2018-19 mid-term reports for Competition 1, and the 2016-18 annual reports for Competition 2

As further illustrated in Figure 14, the distribution of collaborations by sector indicates that, although the majority of collaborations are with individuals from research organizations, Competition 2 funded institutions have engaged with a greater proportion of individuals from private sector and public sector compared to institutions funded in Competition 1. However, it would be premature to draw conclusions on the reason for these differences.

Sources: 2018-19 mid-term reports (Competition 1 grantees) and 2016-18 annual progress reports (Competition 2 grantees)

In both groups, the majority of collaborations involved researchers from other countries (62% for Competition 1 institutions and 53% for Competition 2 institutions).

Just under half (45%, n=385) of the 846 collaborations reported by Competition 1 grantees were new (i.e., did not predate the CFREF grant). Almost half (47%) of reported collaborations existed before the grant and the remaining (8%) were not categorized as either new or existing by the grantees. Of the 385 new collaborations, 82% are collaborators from academic institutions and research organizations, 9% from the private sector, 8% from the public sector and 2% are collaborators from other sectors (e.g., philanthropic organizations). Of the 385 new collaborations, 62% were international (most commonly in the United States, France, Germany, Italy, Switzerland and the United Kingdom).

5.3 Perceived benefits of partnerships and collaborations

The partnerships and collaborations have been pivotal in allowing funded institutions to advance their scientific and institutional strategies. As already documented in subsection 3.2, partners have provided substantial cash and in-kind contributions to support the work undertaken by the funded institutions. As noted during interviews, this has considerably broadened the range of research activities undertaken, and the opportunities for various applications of the research.

More specifically, the range of benefits identified during the evaluation includes the growing visibility and recognition (including international recognition) that some of these partnerships and collaborations can bring, the access to specific infrastructure or expertise, the networking opportunities, and the establishment of a larger pool of stakeholders, which can help sustain the CFREF initiatives beyond the funding period. It is also important to note that some of these partnerships involve multiple CFREF grantees working together to advance their shared research goals and interests. Evaluation findings, particularly interviews, indicate that, to be successful, the collaborations among CFREF grantees must be allowed to emerge organically over-time as opposed to the program mandating a certain time frame.

Along the same logic, partners who were interviewed emphasized the opportunity that the CFREF grants provide to access a centralized multidisciplinary pool of expertise, as opposed to attempting to deal with separate faculties or departments. In at least one case, the presence of this centralized institutional expertise has motivated partners to locate themselves close to the funded institutions to facilitate their collaborations and gain access to the expertise provided by the grantee.

Underlying all the benefits and opportunities identified by key stakeholders is the notion of multidisciplinary research. The evaluation expected to see the foundation and some illustrations of such an approach, and the findings confirm that CFREF is breaking down silos as it reaches complementary fields of expertise that come together to support a scientific strategy. The extent of multidisciplinary research and how multidisciplinarity is defined vary among the funded institutions. Some of the grantees have expanded the concept quite broadly by bridging, for instance, natural and social sciences, while others have built collaborations among disciplines falling within the same broad scientific domain (e.g., collaborations between physics and engineering). In all cases, however, the ability to engage researchers from different expertise within the common framework provided by the scientific strategy is perceived as a key achievement of CFREF. As noted by representatives from funded institutions, this has permanently shifted their organizational culture, and is expected to expand beyond the activities falling strictly under CFREF grant.

5.4 Research infrastructure

The evaluation also explored if grantees perceived the existing infrastructure to be adequate for supporting the success of the CFREF initiative.

There is an expectation that the funding provided through CFREF and CFI will play a highly complementary role in ensuring that research teams can access the infrastructure they require for their projects, including research facilities (e.g., laboratories), as well as equipment and supplies. The range of eligible expenses under CFREF allows funded institutions to direct some of their grant funds toward these infrastructure costs, but larger equipment items (beyond $300,000) and major capital expenditures, such as new buildings and major research facilities, are not expected to be supported through the CFREF grants (Government of Canada, 2018c).

The 2018-19 annual financial reports indicate that approximately 20% of CFREF funding has been directed to infrastructure-related expenses. Approximately $92.6 million (7.4% of total CFREF funding) has been allocated to costs associated with research facilities, and an additional $162.2 million (13% of total CFREF funding) has been allocated to equipment and supplies.

To complement these investments, some grantees have turned to the CFI to secure infrastructure funding. As part of their mid-term reports, Competition 1 grantees were asked to report on the CFI funding they obtained. This data show that these institutions have secured approximately $71 million in CFI grants since drafting the CFREF proposals (as of March 2019).^{Footnote 26}

During interviews, representatives from the funded institutions emphasized how critical CFI funding had been in building their expertise and raising it to a point where they could successfully compete for a grant of the magnitude of CFREF. As such, the CFI support is perceived to be as critical before obtaining a CFREF grant as it is after the grant is awarded. As noted during interviews, the complementarity of the two programs is also reflected in the fact the CFREF grants raise the intensity of the research undertaken, which in turn, allows institutions to take full advantage of their CFI-funded facilities and equipment.

Overall, representatives from the funded institutions indicated during interviews that CFREF and CFI funding combined have allowed them to assemble the research infrastructure they need to pursue their scientific and institutional strategies. The survey of CFREF participants echoed that sentiment: 70% of respondents noted that the state of the research infrastructure was a factor that encouraged them to join their respective CFREF team.

Funded institutions expect to continue seeking funding from CFI to address emerging needs and gaps as their research progresses and their technologies mature, and as infrastructures require update and renewal. Notably, some key informants noted that it had been challenging to use CFI funds awarded to support the implementation of the CFREF grants because there was a gap in time from when they received the CFREF grant to when they received the CFI grant. As a solution, they suggested that a CFI application could be embedded within future CFREF applications, or that CFREF institutions could apply for CFI “off-cycle,” very soon after their CFREF grant is awarded. That would make it easier for grantees to plan and budget for research activities and related infrastructure investments.

6.0 Program design, delivery and cost-efficiency

This last section of the report addresses certain aspect of the program’s cost-efficiency, design and delivery at the granting agency level. In order to identify lessons learned in support of a third CFREF competition and continued program implementation, interviewees were asked to share their perspectives of what could be improved. Areas covered included the competition process, the role of TIPS during program implementation, and the monitoring and reporting requirements.

6.1 Administrative costs of delivering the CFREF program and the grants

The assessment of cost-efficiency included looking at funds spent on administration by TIPS to deliver the program as whole, as well as funds spent on administration by grantees. For the CFREF program as a whole, administrative expenditures were looked at in relation to grant expenditures.^{Footnote 27} Between fiscal years 2015-16 and 2018-19, the program cost less than $1 to administer for every $100 granted, with an average of 52¢ per $100 granted over this four-year period. For comparison purposes, TIPS programs as a whole cost between 9¢ (RSF) and $2.8 (NFRF) to administer for every $100 granted in 2018-19 (overall average of $1.2 for every $100 granted).

Source: Financial administrative data

Factors contributing to the low cost for TIPS administering the program include the large size of the grants and the fact that institutions are responsible for parts of the administration which is complex in nature.

As institutions are responsible for the implementation of the grants, they can allocate some of their grant funds to cover administrative costs. At mid-term, just under one-fifth (18%) of total grant funds spent by Competition 1 grantees had been attributed to administrative expenses, including 12% for compensation-related expenses paid to research administrative support staff (direct costs) and 6% for management and administration of the CFREF grant (indirect costs). The proportion of grant funds spent on administrative expenses by grantees ranged from a minimum of 10% and a maximum above 30%.^{Footnote 28} The overall proportion was the same as what these grantees expect to spend over the entire grant term. There is currently no cap on what proportion of the CFREF grants can be spent on salaries of research administrative support personnel, but in light of these findings, it might be worthwhile to look at if what is spent is in line with TIPS expectations of how funds should be spent overall. However, when considering the percentage of administrative expenditures, it is important to note that these are calculated in relation to the total CFREF grant funds and not in relation to the total budget that each CFREF manages which also includes leveraged funds from the institution and partners. That is, a portion of these leveraged funds may also have been used for administrative expenses by the institutions.

6.2 CFREF grant competition process

Several factors have contributed to making the first two competition processes fairly challenging for all involved. From the perspective of applicants, CFREF was providing what was often described as an unprecedented level of funding and yet, the program gave significant latitude to those applying. While this offers many benefits, interviews with funded and unfunded institutions indicated that many uncertainties surrounded the application process, and that they had not received sufficient supplementary guidance or clarifications. It appears that both those administering the program and the applicants were simultaneously gaining experience in responding to or managing CFREF’s unique features.

In addition to having been implemented within a relatively short timeframe,^{Footnote 29} the application process for Competition 1 and 2 were slightly different, particularly as it relates to the requirement for a Letter of Intent as part of the second competition. Applicants noted that in both competitions, they had limited time to reach out and confirm potential partners and design their proposed management structure for the grant. This is clearly reflected in the fact that many grantees have modified the governance structure that they had initially placed in their submission.

From the perspective of some of those who reviewed funding applications and made recommendations for funding, the process was described as rewarding, particularly in light of the level of funding provided and the opportunities it was providing, but they did offer suggestions for improvement. They emphasized the need for additional time to review these complex applications and to further align the scientific knowledge of the reviewer with the nature of the proposals submitted. They also suggested the need to maintain some communications between the reviewers and TIPS to provide updates on the final funding decisions and the monitoring and assessment activities following the awarding of the grant.

6.3 Role of TIPS during program implementation

After the funding has been awarded, representatives from TIPS and the funded institutions are collaborating to support the grant implementation. During interviews, representatives from funded institutions emphasized the availability, professionalism, and commitment of TIPS representatives, who systematically aspired to provide support in response to questions or requests.

TIPS and the institutions also engage more formally during an initial on-site visit after two years of grant implementation, and during the more formal assessment as part of mid-term review: the latter also involves an expert panel. The evaluation confirms the importance of these two processes, not only in allowing TIPS representatives to better understand the experience of grantees, but for the grantees themselves to help ensure that they are on the right track. In fact, evaluation findings indicate that more sustained monitoring activities would benefit grantees. This could take the form of ongoing communications, and informal visits when feasible, to support an ongoing dialogue that allow TIPS to better understand that is being done at the grant level, and for grantees to have more opportunities to understand TIPS’ expectations. As noted earlier, for most institutions, this is the largest grant they have ever received, and as they proceed to implement it, they would benefit from a greater level of interactions with TIPS representatives.

Along the same logic, the “CFREF summits,” which are organized by TIPS and the CFREF grantees themselves, have also proven to be particularly beneficial. These ad-hoc gatherings where all funded institutions are invited have provided participants with a unique opportunity to share experiences and best practices and explore potential collaborations. As CFREF is planning for a third competition, such gatherings will become ever more helpful to ensure that any new grantees can benefit from the collective experience gained to date.

6.4 Monitoring and reporting requirements

The evaluation provided insights on the experience gained to date with performance monitoring activities related to CFREF grants. In particular, the performance information collected by TIPS offered a valuable support in understanding the range of activities undertaken to date and the progress made by the program towards its immediate outcomes, recognizing that it is too early to assess the extent to which the program’s intermediate and long-term outcomes have been achieved. The detailed mid-term reports and the results from the mid-term review process were particularly helpful for the evaluation in that regard. Considering the size of the grants and the complexity of the scientific and institutional strategies they support, these reports provided important information related to the implementation of the grants and their achievements to date.

Based on the experience gained to date, the evaluation identified potential areas for strengthening monitoring and reporting activities, which are summarized below.

The annual progress reports and the mid-term reports. The evaluation indicates that there are uncertainties, among grantees in particular, as to the purpose and use of annual reports. It was noted, for example, that no feedback was received following the submission of annual reports (on either the adequacy of the report itself or the actual content). As such, some grantees questioned whether they are meeting TIPS’ expectations, or if the content they are providing could be improved. In light of this, it appears that a clearer articulation of the purpose and use of annual reports would facilitate the engagement of grantees in providing this important information.

The analysis of the annual reports completed as part of this evaluation also points to a need to review the reporting templates (both the annual and mid-term reports) to ensure that grantees are asked to report in the same format for common data elements and to clarify some definitions (e.g., what constitutes a new or recruited faculty and HQP, parameters for what publications should be listed as having resulted from the initiative, specific timeframes for other federal funding received related to the scientific strategy).

This would enhance the consistency and comparability of data across different reporting periods, since many elements are included in both the annual progress reports and the mid-term reports. The quality of data collected through these templates would be improved by having TIPS dedicate resources for a detailed review to identify inconsistencies, reporting errors or missing data when these reports are received. In addition, evaluation findings confirm the importance for grantees to be informed early on about the full set of data that will be collected as part of the mid-term report, so that they can prepare accordingly.

Performance Measurement Plans (PMPs). As part of their original proposals, each grantee was asked to develop a PMP, which forms the basis for ongoing reporting, progress-monitoring and the evaluation of results during the mid-term review. The purpose of the PMP is twofold: first, it is expected to track the grants’ progress toward specific performance targets to be achieved by mid-term the grant (i.e., at the end of the fourth year of the award) and by the end of the award period; second, the PMPs are expected to support the collection of performance data at the CFREF program level that will contribute to a better understanding of the overall progress of CFREF in achieving its program-level outcomes. This way, PMPs can help grantees articulate their vision and document their progress and can help the federal government in documenting the achievement of CFREF as a whole.

Notably, there was considerable variability with respect to the type of performance targets and the specificity of performance indicators that were included in the PMPs. At the time of the evaluation, many of the PMPs were not achieving these dual purposes and were largely limited to broad performance indicators that were often not explicitly linked to the transformational logic of the grant itself. While some grantees felt that the PMPs were useful for tracking and monitoring the performance of their grant, others described maintaining the PMP strictly as a reporting requirement for TIPS. During interviews, it became clear that grantees varied in terms of their understanding of the purpose of the PMP; namely, whether they were intended for performance measurement at the grant level (i.e., as an internal tool for tracking progress), for TIPS’ use in documenting and assessing performance at the program level, or both. For instance, many grantees expressed uncertainty about how the PMPs are weighted and considered during the mid-term review process and noted that no feedback was provided by the program on the PMPs in terms of ongoing performance of the grant. Some interviewees noted that receiving such feedback would be helpful to either assure grantees that they are on the right track, or to signal where there might be opportunity for improvement prior to the formal assessment of performance at the mid-term review. Moreover, many grantees expressed a desire to be better supported in developing and refining their PMPs (e.g., having examples of PMPs, and/or increased guidance from TIPS on what types of indicators were appropriate). Some interviewees confirmed that performance measurement was a new undertaking for them, and that they were unsure of how to select relevant and valid performance indicators for their proposed scientific and institutional strategies.

Overall, this evaluation concludes that it would benefit both grantees and TIPS to more clearly articulate the purpose and intended use of performance data collected through the PMPs, and to clarify if performance indicators should reflect both grant-level and program-level targets. A variety of avenues, including the CFREF “summits,” could be considered to allow for sharing experience and lessons learned on developing a PMP that serves the information needs of both the grantees and TIPS. In addition, TIPS and grantees could consider the following to improve the PMPs:

• Include, in each PMP, a clearly articulated description of the transformational logic of the individual grant itself (e.g., through a logic modelling process). Beyond the activities undertaken, this description could cover what the grant is expected to achieve in the short- and long-term (i.e., its post-grant legacy), and the contextual considerations and factors that may facilitate or hinder the achievement of these results. During interviews, a couple of grantees expressed that the highly structured format of the mid-term report did not allow them to communicate the key achievement and impacts of their grants in a way that they felt showcased their accomplishments. The PMPs could help address this concern.
• Include common CFREF -level indicators, but also grant specific indicators based on the grant’s transformational logic, as well as a balance of quantitative and qualitative data, in order to allow the actual story of the grant itself to more clearly emerge.

End-of-grant reporting. At the time of the evaluation, grantees were not expected to produce specific end-of-grant reports. As such, we can assume that each grantee would produce a final annual report, which would cover the last year or two of their respective grant. Such a scenario would likely leave a significant gap in the performance information required to appropriately document the overall achievements of each grant. Considering the key role that mid-term reports have played in supporting the current evaluation, and in view of the fact that another evaluation of CFREF is to be expected in approximately five years, it would be helpful to explore the possibility of having each grantee prepare an actual end-of-grant report that would cover similar themes as those currently included in the mid-term reports, with the appropriate adjustments as applicable. The goal would be to allow each grantee to document their overall experience with their CFREF grant, including the results they have achieved, the key challenges they have faced, and any lessons learned. The exact content of these end-of-grant reports would need to be discussed between TIPS and the grantees. Such information would complement other efforts currently underway to document the achievements of CFREF, such as information on CFREF-related scientific outputs though a bibliometric study.

7.0 Conclusions and recommendations

The purpose of this evaluation was to provide senior management at CIHR, SSHRC and NSERC with an assessment of relevance and performance of the CFREF program, as well as aspects of design and delivery. The evaluation had a particular focus on immediate outcomes of the first five grants awarded, as it was conducted four years into the delivery of the CFREF program; as such, it was too early to assess intermediate and longer-term outcomes and impacts of these grants and for the program as a whole. Similarly, it was too early for this evaluation to assess longer-term expected results of investing at the institutional level, or to conclude if the program’s focus on funding at the institutional level confers specific advantages or disadvantages compared to funding at the researcher or project level.

Relevance of the CFREF program and alignment with government priorities

The CFREF program remains relevant and aligned with many government priorities. It occupies a unique niche in the Canadian funding landscape, and is well aligned with government priorities on innovation, talent recruitment, and EDI. At this time, however, there is underrepresentation of members of the four designated groups within CFREF-funded teams. The program literature is also silent on the role of CFREF, if any, in supporting ECRs.

CFREF occupies a unique niche in the Canadian funding ecosystem as it provides funding at the institutional level and provides a vehicle for the federal government to strategically invest in priority research areas that have the potential to create long-term economic advantages for Canada.

The CFREF program is well aligned with government priorities on innovation, talent recruitment and EDI. As a requirement of the program, all grantees have implemented EDI plans to ensure that individuals from the four designated groups (women, visible minorities, Indigenous Peoples, and persons with disabilities) have an equal opportunity to participate and benefit from the program. The evaluation found that institutions have made some progress in implementing their EDI plans, and are aware that this is an important priority for both the agencies and the federal government. Despite efforts to date, however, program self-identification data indicate that there is an overall underrepresentation of individuals from the four designated groups within the program, particularly among Indigenous Peoples (currently 0.5% of participants) and persons with disabilities (currently 2% of participants). Many grantees acknowledge that there is still work to be done to improve diversity in their CFREF teams and governance structures.

Support for ECRs was first introduced as a priority by the Government of Canada in 2018, three years after the CFREF program was launched. As such, this priority did not influence the program’s design at that time. Currently, TIPS collects information on the number of ECRs involved in the initiatives at mid-term, which suggests that the program’s contribution to this priority is of interest to management, however, CFREF’s objectives and expectations of grantees as it relates to supporting ECRs has not been clearly defined in the program information.

The implementation of CFREF

The evaluation has not identified gaps or shortcomings that would raise reasonable concerns about grantees’ governance structures or capacity of grantees to adequately manage their grants or leverage funding at this point in time. At roughly mid-way through their funding, grantees had spent 23% of their grant funds; the use of funds over time should be carefully monitored going forward, particularly because the COVID-19 pandemic may cause additional delays.

Governance structures vary among funded institutions and the flexibility that CFREF offers grantees to build their own governance structure was identified as a key strength of the program by many key informants. In particular, having a framework by which to engage senior personnel (i.e., a vice-president of research), to establish a dedicated administration team for the grant, and to connect with other grantees were identified as key features that support the strategic direction of the grant. The strategic focus appears to evolve over time and the mid-term review process, involving expert peer-review, has an important role to play in challenging the grantees to demonstrate that the scientific direction of their initiatives is on track to help the institutions become world-leaders in their area.

The range of funding allocation mechanisms used by grantees is fairly traditional (e.g., competitive processes), but the unifying framework of a common research program distinguishes CFREF grants from other funding the institutions and researchers receive. Grantees have, however, experienced some challenges and delays in the start-up phase. In particular, many reported that the first year of the grant was largely spent establishing a detailed implementation plan and putting their governance and funding allocation structures into place.

As of March 2019 (i.e., fourth year for Competition 1 and third year for Competition 2), grantees had spent 23% of the $1.2 billion awarded for their grants. At the time of the evaluation, funded institutions and their partners had committed $1.3 billion in additional funding to support the scientific and institutional strategies of the grantees, another $194 million from other federal programs (other than the CFI) is also supporting these research activities. The delays in the start-up phase, which are common for large-scale funding programs, have led to a need for some grantees to seek and receive no-cost two-year grant term extensions. The use of funds over time should be carefully monitored going forward, particularly because the COVID-19 pandemic may cause additional delays.

Participants, partnerships, collaborations and infrastructure

CFREF has enabled the establishment of new partnerships, and in many cases has also enabled the strengthening of pre-existing partnerships and collaborations. As of March 2019, one-half of the $1.3 billion committed by funded institutions and their partners had been secured.

CFREF-funded activities have engaged more than 6,700 individuals occupying various research or support functions. At the time of the evaluation, the most predominant groups of participants were graduate students (36%), faculty members (23%) and postdoctoral fellows (13%). Of total faculty involved in Competition 1 CFREF grants at the time of this evaluation, 50 were Tier 1 CRCs, 33 were Tier 2 CRCs, and eight were holders of Canada Excellence Research Chairs. Grantees from this competition reported recruiting 57 new faculty members to CFREF initiatives. CFREF participants identified several benefits to participating in the grant, including access to an enhanced interdisciplinary research and training environment, access to state-of-the-art research facilities and equipment, and access to complementary training programs that develop HQP’s non-academic skills (e.g., communication, knowledge translation and commercialization) and employability.

As part of the implementation of their activities, grantees have engaged more than 600 partners and close to 1,500 collaborators at the national and international levels. In approximately half of the cases (for both competition 1 and 2 grantees), these were existing partnerships or collaborations that were allowed to expand through CFREF funded activities. Grantees and those interviewed for the case studies reported that many of the pre-existing partnerships had been strengthened following the recipient of the grants. Just over one-quarter of the partnerships (28%) and more than half of the collaborations (58%) were international. The exact contribution of CFREF in allowing these partnerships and collaborations to emerge or expand cannot be measured precisely, but evaluation findings indicate that receiving grants of the magnitude of CFREF has facilitated this outcome. Although CFREF has no specific requirement for the matching of funds, grantees from both competitions have successfully leveraged a significant level of funding in support of their strategies. As of the end of the 2018-19 fiscal-year, one-half of the $1.3 billion committed by the funded institutions and their partners for the seven-year period covered by the grants had been secured.

For grantees, these partnerships and collaborations have provided growing visibility and recognition at national and international levels and access to wider range of infrastructures, equipment and expertise, both from a scientific and commercialization perspectives. Partners and collaborators have also benefited from the expertise, infrastructure and equipment provided by the grantees, in addition to having a centralized access to the full range of multidisciplinary knowledge offered by the grantees.

As of March 2019, grantees had invested a combined total of $255 million in research facilities, equipment and supplies. The CFI has played a critical role in providing complementary support to ensure that the required infrastructures are available to conduct the funded research. Data from Competition 1 grantees show that these five institutions had secured $71 million since drafting the CFREF proposals up until mid-term.

Program design, delivery and cost-efficiency

CFREF program has a very low cost-efficiency ratio: less than $1 was spent on administration of the program for every $100 of grants awarded.

An analysis of cost-efficiency data suggests that the CFREF program has been delivered by TIPS in a very cost-efficient manner to date. Between 2015-16 and 2018-19, operating expenditures within TIPS to manage CFREF as a whole averaged 52¢ for each S100 dollar granted, which is lower than the overall administration costs for programs administered by TIPS. As for Competition 1 grantees themselves, they had used 6% of their funds to support the management and administration of the CFREF grant (indirect costs), in addition to 12% of their funds for compensation-related expenses for administrative support personnel (direct costs).

Evaluation findings suggest that TIPS administrative costs of delivering the program may be too low for supporting effective implementation and monitoring. The evidence is, however, not conclusive given that it was beyond the scope of the evaluation to conduct a more detailed costing analysis (e.g., activity-based costing).

Communications

Based on their experience, grantees would appreciate having more sustained communications with TIPS during the application phase and the implementation of the grants.

The processes relating to the first two competitions were fairly challenging for all involved. CFREF provided what was often described by applicants as an unprecedented level of funding and the program gave significant latitude to those applying in determining their scientific and institutional strategies. While this is viewed as a strength, it meant that there were many uncertainties surrounding the application process, and applicants (from both funded and unfunded institutions) had to seek supplementary guidance and clarifications from TIPS. Based on their experience, applicants indicated they needed more guidance than they received.

As for the ongoing implementation of the grants, grantees would generally appreciate having more sustained communications and interactions with TIPS. As noted earlier in the report, grantees reported experiencing some challenges in the start-up phase and would have benefitted from additional assistance as they finalized their governance models. Ensuring sustained communications, and conducting informal visits when feasible, would support an ongoing dialogue and provide grantees with more opportunities to ensure that they are on-track well in advance of the mid-term review.

Finally, the opportunity for grantees to connect with each other offers many benefits, and this appears particularly true when it comes to sharing lessons learned on the effective, strategic management of a large institutional grant such as CFREF. The “CFREF summits,” which are organized by TIPS and the CFREF grantees themselves, have proven to be particularly beneficial and have provided participants with a unique opportunity to share experiences and best practices and explore potential collaborations. In the event of a third competition, such forums will become ever more helpful to ensure that any new grantees can benefit from the collective experience gained to date.

Monitoring, reporting and performance measurement

Monitoring, reporting and performance measurement activities are in place for the CFREF program, and these provide useful information on grant activities and progress toward intended outcomes. Several areas for improvement have been identified to improve consistency in reporting and better document achievements and program outcomes.

The performance information collected by TIPS through annual and, in particular, the mid-term reports was valuable for this evaluation, and provided a good overall understanding of the range of activities undertaken by grantees to date, as well as the progress made by the program toward its immediate outcomes. Although useful data are being captured through the annual and mid-term reports, some important areas for improvement were identified. Notably, this evaluation recommends clarifying definitions and wording across the annual and mid-term report templates to ensure greater consistency and quality of data collected though these reports.

Given the size of investment and the unique institutional focus of CFREF, there is immense stakeholder interest in documenting outcomes and understanding the broader impact and contribution of CFREF to advancing Canada’s position on the global research stage, and whether the legacy of CFREF will persist beyond the duration of the granting period. Performance measurement activities are critical in capturing progress and outcomes of the grants, and it is therefore important that grantees’ PMPs are able to document performance in a way that is reliable and useful to grantees, TIPS and mid-term review panels. To this end, the evaluation recommends clarifying the purpose and intended use of performance data collected through the PMPs, encouraging grantees to clearly articulate a long-term vision for what they want to accomplish through their grant (and identify how they will know if they have been successful in achieving their intended outcomes), and ensuring that the PMPs include both grant-level and program-level targets in order to more fully document outcomes. In addition, the mid-term review panel suggested that grantees be asked to make regular updates to their plans describing how transformational changes will be sustained. As such, grantees could be asked to articulate these plans as part of their PMPs, in order to ensure that sustainability is taken into consideration at early stages of the CFREF grants, well in advance of the mid-term.

As a final reporting consideration, the current absence of an end-of-grant report could leave a significant gap in the performance information required to appropriately document the overall achievements of each grant. This evaluation recommends that TIPS explore the possibility of instituting an end-of-grant report that would cover similar themes as those currently included in the mid-term reports. This would allow grantees to document their overall experience with their CFREF grant and to highlight results achieved, the expected legacy or long-term/continuing impacts of the grant, key challenges and any lessons learned. This reporting would also facilitate TIPS’ efforts to document and transparently communicate and report on program outcomes and would provide evidence for future evaluations.

Recommendations

Although it was too early in the program’s lifecycle to assess the longer-term expected results of investing at the institutional level, the program remains relevant, has largely met its immediate outcomes (i.e., governance structures and funding allocation processes within institutions, partnerships, collaboration, attraction/retention of teams) and demonstrated progress toward some of its intermediate outcomes (i.e., infrastructures, training environments). This conclusion is supported both by the data collected for this evaluation and the results from the mid-term review of the first five grants where all grants were approved for continued funding with minor, grant-specific recommendations. It should be noted, however, there remains some question about how the transformational changes brought about by the CFREFs will be sustained. The manner in which grantees described their plans for sustaining the transformative changes largely focused on what other funding would be sought to allow them to maintain the momentum of their research activities once their CFREF grants ended. Although grantees described some activities and early outcomes that are indicative of legacy and point to long-term institutional impacts of the program (e.g., new faculty positions created in areas of the CFREF and enhancements to the training environments), the overall results from the mid-term review and the evaluation suggest that securing funding for sustaining the transformational changes brought by the CFREFs could be a challenge following the end of the granting period.

An analysis of cost-efficiency data suggests that the CFREF program has been delivered by TIPS in a very cost-efficient manner to date, however, evaluation findings suggest that TIPS’ administrative costs of delivering the program may be too low for supporting effective implementation and monitoring. Specifically, grantees and applicants identified some challenges with respect to design and delivery of the CFREF program, some of which could be mitigated by improving communications between TIPS and grantees/applicants. The evaluation also identified strengths and limitations of the current monitoring, reporting and performance measurement activities. Based on these conclusions, the evaluation offers the following recommendations to improve the CFREF program:

Appendix A – CFREF Logic Model

Appendix B – Evaluation Methodology

The evaluation of the CFREF program used multiple lines of evidence that included analyses of secondary data collected by TIPS, as well as primary data collection from stakeholders using qualitative and quantitative methods. The various lines of evidence are described below.

Appendix C – References

Canada Research Chairs. (2018, September 10). Establising equity and diversity targets.

Canada Research Chairs. (2017).

Canada’s Fundamental Science Review Panel. (2017). Investing in Canada’s Future—Strengthening the Foundations of Canadian Research.

Canadian Association of University Teachers (April, 2018). Underrepresented & Underpaid: Diversity & Equity among Canada’s Post-Secondary Education Teachers.

Council of Canadian Academies. (2018). Competing in a Global Innovation Economy: The Current State of R&D in Canada.

Employment and Social Development Canada. (2019). Employment Equity Act: Annual Report 2018.

Government of Canada. (2014a). Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation 2014. Industry Canada.

Government of Canada. (2014b, February 11). Economic Action Plan 2014, The Road to Balance: Creating Jobs and Opportunities.

Government of Canada. (2014c, December 16). Canada First Excellence Research Fund: Program Details. Canada First Research Excellence Fund.

Government of Canada. (2016a). Budget 2016: Growing the middle class.

Government of Canada. (2016b). Canada First Research Excellence Fund: Application Instructions.

Government of Canada. (2016c, October 6). CFREF Competition Statistics.

Government of Canada. (2016d, October 19). Canada Excellence Research Chairs Play Significant Role in Canada First Research Excellence Fund initiatives.

Government of Canada. (2017a). Budget 2017: Building a strong middle class.

Government of Canada. (2017b, March 8). Canada First Research Excellence Fund: About Us. Canada First Research Excellence Fund.

Government of Canada. (2017c, May 10). Canada First Research Excellence Fund: Eligible and ineligible expenditures.

Government of Canada. (2017d, October 4). Canada First Excellence Research Fund: Performance and Financial Reporting.

Government of Canada. (2018a). Budget 2018: Equality and growth, a strong middle class.

Government of Canada. (2018b, April 3). Institutional Eligibility Requirements.

Government of Canada. (2018c, July 23). Canada First Research Excellence Fund Administration Guide.

Government of Canada. (2018d, August 2). Canada Excellence Research Chairs: Application and Nomination Process.

Government of Canada. (2018e, September 4). Innovation Superclusters Initiative: Program Overview.

Government of Canada. (2019). Budget 2019: Investing in the middle class.

Government of Canada. (2020a, January 14). Innovation for a Better Canada.

Government of Canada. (2020b, January 27). Canada’s superclusters.

Government of Canada. (2020c, March 20). New Frontiers in Research Fund: Transformation.

Government of Canada. (2020d, April 17). New Frontiers in Research Fund.

Government of Canada, I. C. (2016e, March 8). Canada First Research Excellence Fund: Application Process.

NSERC. (2015, July 30). NSERC—Contact Newsletter.

NSERC-SSHRC Evaluation Division. (2015). Evaluation of the Networks of Centres of Excellence (NCE) Program.

OECD. (2018). Main Science and Technology Indicators.

Research, Technology & Development Topical Interest Group. (2015). Evaluating Outcomes of Publicly-Funded Research, Technology and Development Programs: Recommendations for Improving Current Practice.

Scherer, S. (2014, July 4). Examining Canada’s scientific footprint. The Globe and Mail.

Statistics Canada (2016). Data Products, Census of Population.

Statistics Canada (2016). University and College Academic Staff System—Full-time Staff (FT-UCASS).

Statistics Canada (2019). Survey of Postsecondary Faculty and Researchers. Table 37-10-0165-01 Selected population characteristics of postsecondary faculty and researchers by region, role, and employment status.

TIPS. (2017a). Canada First Research Excellence Fund—Annual Progress Report instructions.

TIPS. (2017b). Performance Measurement Strategy for the Canada First Research Excellence Fund (CFREF).

TIPS. (2017c). Equity, Diversity and Inclusion Performance Measurement Strategy.

Date modified:

2021-07-13