Prepared by:
SSHRC/NSERC Evaluation Division
PRA Inc.

© Her Majesty the Queen in Right of Canada, represented by the Minister of Innovation, Science and Economic Development Canada, 2020

Cat. No. CR22-110/2020E-PDF
978-0-660-35639-6

PDF document

---

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Evaluation background and purpose
  • 1.2 Evaluation scope and questions
  • 1.3 Evaluation methodology
• 2.0 Relevance
  • 2.1 External factors influencing the need for CERC and C150 programs
  • 2.2 Niche of CERC and C150 programs in Canada
• 3.0 Attracting world-class researchers to Canada
  • 3.1 Calibre of CERC and C150 chairholders
  • 3.2 Diversity of CERC and C150 chairholders
  • 3.3 CERC core team members
  • 3.4 Diversity of CERC core team members
• 4.0 Enhanced research capacity at Canadian universities in strategic areas of importance to Canada
  • 4.1 CERC chairholder performance since award
  • 4.2 Institutional growth
  • 4.3 Collaborations and partnerships
  • 4.4 Research outputs
• 5.0 Sustained research capacity at Canadian universities in strategic areas
  • 5.1 CERC chairholders remaining at host institution
  • 5.2 Retention of CERC core team members
  • 5.3 Retention of collaborations and partnerships built through the CERC program
• 6.0 Design, delivery and efficiency
  • 6.1 Overall design and delivery
    • 6.1.1 EDI-specific elements of design and delivery
  • 6.2 Cost-efficiency
• 7.0 Key findings and recommendations
• Appendix A – Program profiles
• Appendix B – Evaluation matrix
• Appendix C – Evaluation methodologies
• Appendix D – References

---

Executive Summary

About the Funding Programs

Launched in 2008, the Canada Excellence Research Chairs (CERC) program supports Canadian universities in research and innovation with an award of $10 million over seven years to facilitate the attraction of world-class researchers to become CERC chairholders in areas of strategic importance to Canada. These chairholders build core teams at their host institution for the purpose of developing and expanding research programs in their respective areas of study.

Launched in 2017, the Canada 150 Research Chairs (C150) program aimed to attract top-tier, internationally based scholars and researchers to Canada (including Canadian expatriates) in order to commemorate Canada’s 150th anniversary. Open to researchers of all disciplines and career stages, the program offered a one-time investment to Canadian institutions of either $350,000 or $1 million per year for seven years with the ultimate goal being to further Canada’s reputation as a global centre of research excellence.

About the Evaluation

The scope of the evaluation covers the period from 2013-14 to 2017-18 for the CERC program, and from 2017-18 to 2018-19 for the C150 program. The purpose of this evaluation is to provide program Management and Steering Committee members with key information on the relevance and performance of the CERC and C150 programs. Given that C150 chairholder positions were awarded in 2018 and 2019 (overlapping with this evaluation), the assessment of this program was more limited in scope than that of CERC. We examined both programs’ relevance; contributions to attracting world-class researchers to Canada; and aspects of design, delivery and efficiency. The evaluation of the CERC program additionally considered outcomes of interest, including an assessment of the extent to which CERC contributed to building and sustaining research capacity in Canada within the strategic areas identified by the federal government.

The evaluation of the CERC and C150 programs features multiple lines of evidence, including a review of program documents and key literature, a review of files and administrative data, a bibliometric analysis, case studies of a sample of CERCs from the first cohort, a survey with CERC core team members and key informant interviews. Interviews were conducted with CERC chairholders (those not included in case studies, those who left before the end of their term, and active Competition 2 chairholders), C150 chairholders (including those who declined the award), vice-presidents (VPs) of research, and selection committee and review panel members.

Primary limitations of the data were the small sample of chairholders (N = 26 CERC chairholders from Competitions 1 and 2; N = 25 C150 chairholders), which precluded the use of inferential statistics in the context of certain analyses. Despite some missing data associated with the survey of core team members and CERC annual reports, we are confident in our ability to report major trends from the quantitative data—particularly given our ability to triangulate these quantitative findings with other lines of evidence.

Conclusions and Recommendations

Evaluation Question 1: To what extent do the CERC and C150 programs continue to address a unique need?

Key findings suggest that CERC and C150, namely due to their prestige and value, are unique in their ability to attract and support world-class international researchers in building research capacity within Canada. These programs represent a specific niche in federal funding programs. Despite a few institutional representatives expressing concern about the tension created among faculty as a result of the disproportionately high level of funding issued to a single research team, most stakeholders shared positive feedback about CERC: they commonly expressed that the program was necessary to attract the calibre of researcher in question. Overall, the perception among stakeholders, combined with the influx of world-class researchers into Canada and their noted productivity thus far, is that Canada should continue investing in scientific research through CERC and C150. Other countries are making large investments in research; therefore, in order to remain globally competitive, Canada needs to continue offering awards of similar calibre.

Evaluation Question 2: To what extent have the CERC and C150 programs attracted world-class researchers to Canada?

Through bibliometric analysis, this evaluation has determined that the CERC and C150 programs have indeed been successful in attracting world-class researchers to Canada. In turn, the reputation and innovative research of these chairholders has been cited as a main factor in attracting faculty and highly qualified personnel (HQP) to core teams, and has also facilitated the forming of partnerships and collaborations both nationally and internationally.

Evaluation Question 3: To what extent have the CERCs contributed to enhanced and sustainable research capacity at Canadian universities in areas of strategic importance identified by the federal government?

In the context of this evaluation, sustainability is operationalized as the growth and retention of the core team, partnerships and collaborations built through the CERC, and the continued prolific production of quality research outputs. An in-depth assessment of sustainability capturing longer-term impacts will likely only be possible in the context of the next CERC evaluation. However, it was still possible to gauge sustainability in a preliminary way in this evaluation by examining chairholders’ and core team members’ intent to remain at their host institution and/or in Canada, as well as the perceived impact of the end of CERC funding on partnerships and collaborations.

The majority of chairholders perceive the CERC program to have had great influence on their ability to establish both national and international partnerships and collaborations, which in turn have been useful in leveraging additional sources of funding and laboratory resources. Bibliometric analysis has found that CERC host institutions have seen significant increases in annual publications in the research area of the CERC as a direct result of the chairholders’ output (approximately an additional 10 articles per year or a 13.3% increase). The CERC host institutions are also well above their comparator Canadian and foreign institutions in annual publications. Although the measured increase in publications from pre- to post-award for individual chairholders is modest rather than pronounced (and commensurate with the relative increase observed among Canada Research Chairs [CRCs]), these figures are likely an underestimate given that (1) CERC-funded research outputs (even from Competition 1 CERCs) are still emerging and (2) these data only capture the chairholder’s publications and not those of the CERC team as a whole. Finally, publication volume captured through bibliometric data is but one of many indices of research capacity and contributions.

Although it might be of interest to examine how the CERC program compares to other tri-agency programs with respect to cost per publication, such comparisons would likely result in misleading conclusions. Recall that a CERC is defined not only by the chairholder’s contributions, but more holistically by the core team and research program that is built at the host institution, as well as the larger research networks that the CERC team establishes. Qualitative data from the current evaluation suggest that the CERC program has resulted in increased research capacity at host institutions and has greatly influenced the career trajectories of team members, thus contributing to a range of successes that extend beyond the accomplishments of the chairholder alone. Although the broader, cascading impacts of the program could not be quantified, it is important to keep these larger contributions in mind when comparing the CERC program to other programs that may, by design, have a narrower reach.

Importantly, CERCs reported that their partnerships and collaborations are providing linkages primarily with other academic institutions rather than other organizations in the private and public sector. Additionally, CERCs reported a low prevalence of research outputs tailored to government and public policy contexts, primarily citing research outputs tailored to academic audiences (i.e., scholarly refereed journals and conference proceedings). The implication is that CERCs may not be reaching wider audiences beyond academia—an expected intermediate outcome as per the program’s logic model. Although this may in part be due to the longer time period required for government and public policy uptake, this evaluation indicates that increasing the visibility of the chairholders and their research would be beneficial, in turn introducing potential opportunities to establish linkages in other sectors and disseminate research to non-academic audiences, including government decision-makers.

In terms of sustaining research capacity in Canada, the majority of CERC chairholders (nearly 80%) plan to remain at their host institution following their CERC term; in addition, 50% of core team members surveyed indicated a desire to remain in Canada after the CERC term. CERC chairholders’ desire to stay at their host institutions is influenced by a number of factors, including the success and strength of the research program they have created, the investment in infrastructure they have made at the institution, the support and level of commitment to sustainability received from their host institution, and their ability to secure additional funding at the end of the CERC term. Host institutions report growth through the CERC program—namely evidenced by the CERC’s role in the development of new research programs, the creation of new faculty positions, the promotion of research more broadly and the development of new technologies. An additional key concern held by the majority of chairholders was in the overall sustainability of the CERC program and the potential impact that the end of CERC funding might have on their ability to sustain the collaborations and partnerships they have fostered through their position. The degree to which the program truly leads to the creation of sustainable research capacity should be more evident at the time of the next evaluation, at which time over five years will have elapsed from the end of the Competition 1 CERC terms.

Evaluation Question 4: To what extent are the design and delivery of the CERC and C150 programs effective and cost-efficient?

Several key strengths of the programs were noted by both chairholders and institutional representatives, including the flexibility in the use of CERC funds, the value and prestige of the CERC and C150 awards and the selection of the CERCs in strategic areas of research for Canada. However, interviews with chairholders and institutions alike revealed there was sometimes misalignment of expectations, largely a product of proposals and sustainability plans that lacked concrete goals and commitments.

Other concerns raised by chairholders and other key informants surrounded the length of the CERC term; that is, the number of years available to spend the $10 million award. Beyond the fact that the CERC is not renewable, several chairholders indicated that the seven-year term was too short a period to build such a large research program. It was also noted that research needs and timelines vary according to research area. Delays in getting research labs running at the start of the CERC terms was a challenge, especially when considering the high level of progress that is expected with this kind of program. Extending the terms or having a more explicitly defined tapering-off period would be helpful. Although automatic extensions of one year are provided and the Terms and Conditions of the award allow for the possibility of an additional extension with proper justification, the latter option was not universally understood among chairholders and institutions.

The timeframe for the CERC application and nomination process was considered too lengthy and onerous, which ultimately led to the loss of desirable candidates in favour of other job opportunities. In addition to the poor timing of the competition (i.e., over the summer while many people were away and difficult to reach), the primary issue with the C150 competition was the fact that its timeline was too compressed, which created a number of logistical issues and ultimately resulted in candidates declining the potential nomination due to the timeframe.

CERC chairholders from the first and second competitions are relatively homogeneous, generally not identifying with any of the four designated groups (i.e., women, persons with disabilities, Indigenous Peoples and members of visible minorities). However, several advances in the design of the CERC program (that were also applied to the C150 program) have been made since the first two CERC competitions to increase the level of equity and diversity within the program. Namely, the introduction of formal equity, diversity and inclusion (EDI) requirements in the selection criteria and institutional recruitment process, the inclusion of a detailed equity plan and the inclusion of an individual with EDI-related expertise on peer review panels. These changes to the recruitment and selection processes in the latest CERC and C150 competitions have resulted in a more diverse group of chairholders and core team members in terms of an increased representation of women and visible minorities among awardees.

Despite advances over the last few years, there are still a number of EDI implementation challenges, which in part pertain to the lack of clarity around EDI requirements and what recruitment targets should be applied across the various equity-seeking groups. Review panel members reported struggling with how to assess and weigh EDI considerations in the selection and review process. This was also a commonly perceived challenge of chairholders when recruiting core team members. In addition, review panel members expressed that the individual(s) invited to provide EDI-related advice was not used effectively within the peer review process. Finally, institutions lamented the overall lack of clarity regarding the required elements of an equity plan at the time of application.

Performance Reporting

As the CERC and C150 are relatively new programs, reporting practices have continued to evolve over time. Indeed, a tri-agency working group was formed in 2016 to further refine annual reporting templates. Although quantitative data extracted from annual reports were sufficient to support the evaluation of the program, the formulation and structure of key questions were often modified from year to year, which in many instances precluded longitudinal analysis. Additionally, based on wide variability in responses to certain items on the annual report combined with informal discussions with CERC chairholders and their administrative staff, there appeared to be a lack of universally applied definitions for key constructs (i.e., partnerships vs. collaborations; core team member; “providing expert advice”). Finally, there was a general perception by chairholders and institutional representatives that the annual reporting requirements were fairly onerous and lengthy, and that not all collected information was examined.

---

1.0 Introduction

This report presents key findings, conclusions and recommendations from the 2018-19 evaluation of two tri-agency funding programs: Canada Excellence Research Chairs (CERC) and Canada 150 Research Chairs (C150).

1.1 Evaluation background and purpose

The CERC program supports Canadian universities in research and innovation with an award of $10 million over seven years, facilitating the attraction of world-class researchers^{Footnote 1} to become CERC chairholders in areas of strategic importance to Canada. Institutions must ensure 100% in matching funds over the term of the award (excluding tri-agency and CFI sources). Chairholders build core teams at their Canadian host institutions for the purpose of developing and expanding research programs in their respective areas of study.

The C150 program was intended to commemorate Canada’s 150th anniversary and offered a one-time investment to Canadian institutions of either $350,000 or $1 million^{Footnote 2} per year for seven years with an aim to attract top-tier, internationally based scholars and researchers (including Canadian expatriates) from all disciplines and career stages. With the objective of further strengthening Canada’s research capacity and enhancing its reputation as a global centre of research excellence, the program was developed to build on the gains and contributions of other tri-agency programs including CERC, Canada Research Chairs (CRC) and Canada First Research Excellence Fund (CFREF).^{Footnote 3}

The purpose of this evaluation is to provide program Management and Steering Committee members with key information on the relevance and performance of the CERC and C150 programs. Aspects of the program design, delivery and cost-efficiency are also covered. The CERC and C150 evaluations have been conducted in compliance with the requirements stipulated in the Treasury Board’s Policy on Results, with respect to Section 42.1 of the Financial Administration Act, and aligned with the federal government’s commitment to equity, diversity and inclusion (EDI).^{Footnote 4} Please note that although the EDI requirements were embedded within the C150 competition, these policies were not formally applied within the CERC program until the third and most recent competition. Although Competition 3 of the CERC program is discussed to contextualize recent advances in EDI and other processes, it is not formally included in the current evaluation.

1.2 Evaluation scope and questions

The scope of the evaluation covers the period from 2013-14 to 2017-18 for the CERC program, and from 2017-18 to 2018-19 for the C150 program so as to maximize the amount of information available on the latter. The evaluation of C150 was undertaken in tandem with the evaluation of CERC due to the level of commonality between the two programs. More specifically:

• given their similarly large disbursements, C150 was considered a good comparator to CERC in its potential for attracting world-class chairholders;
• the design of the recruitment process for both programs targets international candidates;
• it was deemed more appropriate to evaluate the C150 program with the CERC program in 2018-19, and subsequently five years from now (instead of evaluating C150 with the CRC program two years from now, as was the original plan). Evaluating C150 and CERC in tandem would not only provide early findings on the attraction of world-class researchers and lessons learned related to design and delivery, it would also offer an opportunity to better capture the achievement of program outcomes (and introduce potential adjustments) before the C150 funding period ends;
• finally, given the infancy of C150 and the potential for targeting both CERC and C150 chairholders during the same data collection effort, it was deemed most efficient to evaluate the two programs in tandem rather than evaluate the C150 program separately.

Given that C150 chairholder positions were awarded in 2018 and 2019 (overlapping with this evaluation), the assessment of this program was more limited in scope relative to CERC. The evaluation examined both programs’ relevance in the context of the current national and international research climate; contributions to attracting world-class researchers to Canada; and aspects of design, delivery and efficiency. The evaluation of the CERC program additionally included an assessment of outcomes; namely, the extent to which the program contributed to building and sustaining research capacity in Canada within the strategic areas identified by the federal government. This evaluation included only a narrow selection of CERC program outcomes so as to avoid duplication with the previous evaluation of the program conducted in 2013-14.^{Footnote 5}

The current evaluation was designed to address the following four questions:

• To what extent do the CERC and C150 programs address (or continue to address) a unique need?
• To what extent have the CERC and C150 programs attracted world-class researchers to Canada?
• To what extent have the CERC chairholders contributed to enhanced and sustainable research capacity at Canadian universities in areas of strategic importance identified by the federal government?
• To what extent are the design and delivery of the CERC and C150 programs effective and cost-efficient?^{Footnote 6}

The complete evaluation matrix that aligns the evaluation questions to indicators and data sources appears in Appendix B.

1.3 Evaluation methodology

SSHRC evaluators and an evaluation consulting firm (PRA Inc.) collaborated to design and implement this evaluation. The process was guided by an evaluation advisory committee comprised of representatives from the SSHRC Evaluation Division, the Tri-agency Institutional Programs Secretariat (TIPS), the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canadian Institutes of Health Research (CIHR) and the Canada Foundation for Innovation (CFI).

The evaluation of the CERC and C150 programs features multiple lines of evidence, including a review of program documents and key literature, a review of files and administrative data, interviews with key informants (n = 51),^{Footnote 7} case studies of CERCs awarded during the first competition (n = 9),^{Footnote 8} a web based survey with telephone follow-up of CERC core team members^{Footnote 9} (n = 562,^{Footnote 10} response rate = 37%) and a bibliometric analysis.^{Footnote 11}

The bibliometric study, performed by Science-Metrix and designed in consultation with SSHRC, had two main foci: (1) to measure the CERC and C150 programs’ ability to attract world-class researchers to Canada; and (2) to measure contributions of CERCs to Canadian institutions with respect to research output and impact. Contributions to research output and impact were based on a number of bibliometric indicators yielded for the period following respective award dates.^{Footnote 12}

In terms of measuring the programs’ ability to attract world-class researchers, several bibliometric indicators were analyzed over the 10-year period pre-award relative to a number of comparison groups, including unsuccessful applicants to the programs and matched groups of Canada Research Chairs (CRCs).^{Footnote 13} Although the CERC program issues relatively fewer awards that are larger in value than the CRC program, both programs target established researchers who are acknowledged to be world leaders in their respective fields and include the recruitment of internationally based researchers. As such, the latter was identified as a reasonable and valid comparator.

This evaluation’s key methodological challenges and corresponding mitigating strategies included the following:

1. Reporting practices associated with CERC annual reports have not been consistent over time. Specifically, the formulation and structure of key questions were often modified from year to year, which in many instances precluded longitudinal analysis. This issue was compounded by some missing data in the annual reports, either because chairholders or institutions opted not to answer certain questions or, because under exceptional circumstances identified on an ad hoc basis, a chairholder was not required to submit an annual report in a particular year. Moreover, the nature of the reporting template is structured in such a way that aggregate values of certain fields (e.g., number of core team members, number of partnerships) could not be determined by summing across years given the possibility of double-counting. As such, it was typically only possible to determine the median for a construct of interest at a given point in time; note that we generally opted to report median over mean as a measure of central tendency to minimize the impact of distributional outliers. Finally, the default entry of “0” in later reporting templates resulted in an inability to determine whether “0” values referred to missing data or to the genuine absence of activity. Despite limitations associated with CERC reporting templates and the resulting administrative data for CERC, we were nonetheless able to determine major trends and triangulate findings with alternative lines of evidence to confirm their validity.

   It should be noted that a tri-agency working group was formed in February 2016 to review the logic model (LM) and performance measurement strategy (PMS) of CERC in response to the recommendations of the fifth-year program evaluation. In the context of this working group, TIPS policy and CERC program teams began to redesign the annual report templates for chairholders and institutions, which received final approval by the Associate VP of TIPS in January 2018. Notably, these redesigned templates were used for the 2016-17 and 2017-18 reporting periods and already address many of the reporting issues identified above.

2. Based on wide variability in some of the quantitative data associated with annual reports, combined with informal discussions with CERC chairholders and their administrative staff, there appeared to be a lack of universally applied definitions for certain key constructs. For example, when the evaluation team requested complete CERC core team member lists from chairholders for the purpose of constructing a survey sample frame, it became clear that the definition of “core team member” varied from chairholder to chairholder. Regarding the annual reports, we strongly suspect that respondents have inconsistent definitions for terms like “partnership” and “collaboration” (e.g., the number of collaborators and partners reported by chairholders ranged from 1 to 160 and from 1 to 50, respectively). To address this concern, major trends are often reported over specific numbers where validity is questionable, coupled with triangulation with other available lines of evidence.
3. Complete lists of names and contact information for the CERC core team membership are not currently collected by TIPS. Therefore, as indicated above, the SSHRC Evaluation Division contacted individual chairholders in an attempt to obtain a representative survey sample frame of current and former CERC core team members. In total, 23 of the 26 CERCs provided such lists. Notwithstanding missing data, all CERC core teams had at least some members included in the overall sample given the combination of these data with the sample of core team members who consented to be contacted for evaluation purposes upon completing their self-identification form in June 2018 (a data collection effort managed by TIPS). The impact of missing data on the evaluation findings is expected to be minimal given the acceptable response rate to the survey of 37% (a rate typical of similar evaluations) and representation from each CERC team among the respondent pool.
4. Due to the small number of chairholders in the samples (for both CERC [N=26] and C150 [N=24]), it was not always possible to conduct inferential statistical tests. In the CERC administrative data review and in some bibliometric analyses, only descriptive data were presented to illustrate group differences (e.g., number of CERCs in different priority areas, representation among EDI groups, etc.). Furthermore, certain sub-groups of survey respondents and chairholders could not be disaggregated and analyzed separately given the small number of individuals falling into those categories and because of the potential for identification (e.g., membership to EDI groups, host institution, etc.). However, with the presentation of descriptive statistics and triangulation with other lines of evidence, we can be reasonably confident in the validity of the reported conclusions.
5. Although we compare CERCs, C150s and CRCs in the bibliometric analysis, it should be noted that it was only possible to capture the research productivity of the CERC chairholder and not members of the CERC core team. As such, the bibliometric output is an underestimate of the true productivity of the CERC team. That said, it is reasonable to assume that, with respect to peer reviewed publications, the majority of core team members would list the chairholder as a co-author. Accordingly, bibliometric data are still considered a reasonable indicator of publication output.

---

2.0 Relevance

2.1 External factors influencing the need for CERC and C150 programs

Canada is generally regarded as a strong research country, but over the last decade has begun to lag behind certain countries in research and development (R&D) investment as a percentage of Gross Domestic Product (GDP), research output (e.g., volume of publications in peer-reviewed journals) and research impact (e.g., Average Relative Citation [ARC] score^{Footnote 14}). Emerging countries such as China and India are investing more in research than Canada and a number of other traditionally strong research countries, which is resulting in increased global competitiveness with respect to research output, research quality and the attraction of researchers (Advisory Panel on Federal Support for Fundamental Science, 2017; Council of Canadian Academies, 2018; Government of Canada, 2012). In this context, where countries are investing considerable funding in research, interview and case study findings with institutional representatives at Canadian host universities (e.g., representative from the university research office, dean of faculty associated with the CERC) suggest that the international competitiveness for top talent both within academia and industry speaks to the need and relevance of the CERC and C150 programs.

Additionally, interview and case study findings strongly indicate that recent political changes in the international context have provided an opportunity for Canada to attract world-class researchers. The primary factors articulated by stakeholders include the Trump presidency in the US and Brexit in the UK—two political situations that are resulting in a “brain drain” from those countries and the desire among some researchers to relocate to other countries to pursue their work, with Canada being a strong and viable option.

2.2 Niche of CERC and C150 programs in Canada

Filling a similar niche, the CERC and C150 programs are uniquely positioned to enhance Canada’s status as a world leader in research. Key informant interviews strongly support the view that there are no other funding initiatives in Canada that specifically target and support world-class international researchers and their teams to establish ambitious research programs at Canadian universities. There was general consensus among key informants that the main factors attracting world-class researchers are the level of funding of the CERC and C150 programs and the opportunity to conduct innovative research.

In addition to the CERC and C150 programs, Canadian institutions can benefit from a number of funding opportunities, including the CRC program, the CFREF program, and awards from the CFI. Overall, key informants agreed that the CERC and C150 programs are unique although complementary to other sources of funding such as the CFREF program and CFI programs.

The CERC program is unique

The multiple funding opportunities directed at Canadian institutions have been designed to fill different roles. While the CERC, C150 and CRC programs are institutional programs that help support specific researchers and their research programs, CFREF is an institutional multi-year operating grant. CFI, on the other hand, provides funding for research infrastructure (Advisory Panel on Federal Support for Fundamental Science, 2017).

CERC vs. C150. The emphasis with CERC is on building sustainable research capacity in the area of the CERC at the host university (including training a strong core team), establishing networks and collaborations and providing expert advice to sectors outside of academia. Contrary to CERC, which entails a lengthy multi-stage application process (including both institutional and chairholder applications and review processes), a “rapid response” process was used to facilitate the recruitment of esteemed researchers in the C150 competition—primarily due to the one-time only C150 competition intended to mark the celebration of Canada’s 150th anniversary. In addition, in contrast to CERC’s focus on recruiting established researchers, the C150 program accepted applications from exceptional researchers at all career stages. Finally, Competition 1 and Competition 2 CERCs are almost entirely from the natural or health sciences (aligned with the federal government’s Science and Technology [S&T] Strategy and associated priority areas of research), whereas C150 has greater breadth in its representation from all disciplines of research in the social sciences and humanities, natural sciences and engineering, and health and related life sciences.

CERC vs. CRC. CERC and CRC are currently the major ongoing sources of federal funding for researcher salary support in Canada. These two funding opportunities are often compared because of their similar focus on the attraction of world-class researchers (Advisory Panel on Federal Support for Fundamental Science, 2017; Goss Gilroy Inc., 2016; Science-Metrix, 2010, 2014). The two programs are often perceived as complementary and present many similarities (Goss Gilroy Inc., 2016; Science-Metrix, 2010, 2014). Specifically, both programs support Canada’s global reputation, relying on local or regional concentration of resources to foster specialization (Advisory Panel on Federal Support for Fundamental Science, 2017). Both CERC and CRC programs additionally allow Canadian institutions to create research opportunities both to retain Canadian researchers and attract outstanding international researchers (Tri-agency Institutional Programs Secretariat, 2014).

Beyond certain similarities in their overall mandates, the CRC and CERC programs differ in several respects. Aligning specifically with the strategic areas identified by the federal government, the CERC program was created to further support Canada’s increasing global reputation in research,^{Footnote 15} and strengthen Canada’s ability to attract the world’s top researchers (Advisory Panel on Federal Support for Fundamental Science, 2017; Tri-agency Institutional Programs Secretariat, 2017a). At $10 million over seven years, the CERC award value is much larger than CRC’s award values of either $200,000 annually for seven years (Tier 1) or $100,000 annually for five years (Tier 2). The three consecutive CERC competitions resulted in 40 research chairs (29 awarded chairs^{Footnote 16} from Competitions 1 and 2, and an additional eight awarded chairs [and three pending chairs] from Competition 3). In addition, the link to federal policies and priorities is very strong as the CERC program is guided by federal priorities in science, technology and innovation. The federal government thus determines the S&T priority areas for which these chairs should be awarded (Tri-agency Institutional Programs Secretariat, 2017a). In contrast to CERC, the CRC program awards a larger number of chairs, with a total of 2,285 chair positions allocated to eligible institutions and 1,836 active chairs as of June 2019. Priority areas for the CRC program are determined by individual institutions given that nominations must be aligned with the Strategic Research Plans (SRPs) of each institution (Government of Canada, 2018).

The CERC program has a more global talent emphasis than the CRC program. All CERC chairholders awarded thus far have come from abroad (Goss Gilroy Inc., 2016). Although the CRC program also aims to recruit international researchers, the 15-year CRC program evaluation showed that the percentage of international recruits was declining: whereas foreign nominees respectively accounted for 32% and 31% of new Tier 1 and Tier 2 nominees over the 2005-09 period, they represented only 13% and 15%, over the 2010-14 period. The CERC and C150 programs, aimed at international recruitment, have helped to address this issue.

CERC vs. CFREF. The CFREF award emphasizes institutional specialization and encourages inter-institutional and international collaboration. Indeed, support for institutional initiatives, the creation of partnerships, and the creation of a research-inducing environment are some of the program’s main objectives (Advisory Panel on Federal Support for Fundamental Science, 2017; Tri-agency Institutional Programs Secretariat, 2017b). While the CERC program also aims to achieve these results, the focus is not on institutions, but on chairholders and their teams. As described in the CERC program’s logic model (see Appendix A), the attraction of world-class researchers and high-calibre teams (immediate outcomes) is expected to trigger the growth of universities in strategic areas (intermediate outcome) (Tri-agency Institutional Programs Secretariat, 2017a).

Research funding opportunities are complementary and synergistic

Despite the unique need addressed by CERC and C150, these programs are also complementary to and synergistic with other federal funding opportunities. Based on available data from annual reports, we know that at least half of the CERCs have reported CRCs among their core team members: this proportion is likely an underestimate given the fact that data on CRC participation was only collected in the 2014-15 fiscal year. Moreover, institutions often align their CERC and CFREF applications so that they are in similar research areas. About three-quarters (20 of the 26) of the CERC chairholders from the first two competitions are directly involved in about three-quarters (13 of the 18) of the CFREFs. Additionally, four of the CFREF initiatives are led by CERC chairholders.

By further enhancing and building upon Canada’s reputation as a global centre for science, research, and innovation excellence, the C150 program builds on the achievements made by the CRC and CERC programs for capacity development, and on the CFREF program in terms of global research leadership (Tri-agency Institutional Programs Secretariat, 2017c). Canadian institutions can benefit from the C150 program to extend an existing research area, but can also build a critical mass in a new area (Government of Canada, 2017).

Finally, CFI research infrastructure funds are also an important source of funding for the CERC teams. Nearly all active CERCs (85%) obtained infrastructure support at some point during their CERC term through the John R. Evans Leaders Fund (JELF) or Innovation Fund (IF) funding opportunities offered by CFI; six CERCs obtained CFI funding for more than one project.

---

3.0 Attracting world-class researchers to Canada

3.1 Calibre of CERC and C150 chairholders

Successful CERC and C150 applicants are more prolific in terms of publication rates than unsuccessful applicants in the pre-award period

Overall, bibliometric analyses indicate that when a peer review committee involved in the chair selection process is faced with an almost uniformly high-performing pool of CERC and C150 candidates, field of study notwithstanding, committee members tend to select those with higher research outputs. Specifically, when comparing bibliometric indicators among successful and unsuccessful CERC and C150 candidates over the 10-year pre-award period, successful CERC and C150 candidates were more prolific than unsuccessful candidates based on publication rate (9.9 vs. 6.8 publications per year for the CERC program and 7.2 vs. 5.8 publications per year for the C150 program).

Scientific performance of CERC, C150, and CRC chairholders pre-award is above global levels

Bibliometric indicators revealed that CERCs, C150s, and respectively matched CRCs performed above global levels on a number of citation metrics over the 10-year period pre-award. Congruent with the intention of CERC and C150 to attract world-class researchers, chairholders associated with these programs performed either on par or better on a number of metrics than matched groups of foreign and Canadian CRCs^{Footnote 17 Footnote 18}  (see Table 1 and Table 2). These findings indicate that, although all three programs attract some of the highest calibre researchers in the world, CERCs and C150s clearly distinguish themselves as world-class by bibliometric standards.

Table 1: Scientific performance of active CERC chairholders and matched foreign and Canadian CRC Tier 1 chairholders pre-award

Metric	CERC chairholders (N = 26)	Matched CRC1 chairholders (foreignFootnote 19) (N = 26)	Matched CRC1 chairholders (CanadianFootnote 20) (N = 95)
Publications per year	11.0*	6.6	6.5
Average of Relative Citations (ARC)Footnote 21	2.64*	2.03	2.02
Highly Cited Publications (HCP)10%Footnote 22	29.2%*	22.4%	21.8%
Highly Cited Publications (HCP)1%	4.6%	3.1%	2.9%

*Statistically significant at p < 0.05.^{Footnote 23}

Source: Bibliometric analysis performed by Science-Metrix.

---

Table 2: Scientific performance of successful C150 applicants and matched foreign and Canadian CRC chairholders pre-award

Metric	Successful C150 applicants (N = 24)	Matched CRC chairholders (foreign) (N = 24)	Matched CRC chairholders (Canadian) (N = 96)
Publications per year	7.2*	2.5	4.3
Average of Relative Citations (ARC)	2.61*	2.05	2.23
Highly Cited Publications (HCP)10%	30.9%	28.7%	25.9%
Highly Cited Publications (HCP)1%	6.2%*	2.5%	3.7%

*Statistically significant at p < 0.05.

Source: Bibliometric analysis performed by Science-Metrix.

---

Comparing bibliometric standings of successful CERC and C150 chairholders indicates that both funding streams attracted candidates of roughly equal calibre, a perception that is generally echoed by institutional representatives participating in key informant interviews. Notable differences are only seen on two indicators: average yearly publication output and share of HCP papers. Specifically, recommended C150 candidates have a higher share of papers that fall among the very top cited publications in their field when compared with recommended CERC candidates (6.2% for C150s vs. 4.9% for CERCs). However, on average, recommended CERC candidates published about two more papers per year than C150s during the pre-award period.^{Footnote 24}

3.2 Diversity of CERC and C150 chairholders

Within the group of CERC chairholders from Competitions 1 and 2, there was a lack of representation of individuals who self-identified as belonging to one or more of the four designated groups. Please note that data on the diversity of chairholders from the first two CERC competitions cannot be shared publicly given the low proportion of individuals self-identifying within each of the four designated groups.

Primarily in response to the almost exclusive male composition of chairholders from the first two CERC competitions, along with the recognition of the importance and benefits of EDI to achieving research excellence (i.e., through greater innovation and diversity in perspectives; Hewlett, Marshall, & Sherbin, 2013; SSHRC, 2019), multiple changes were introduced by TIPS in the third CERC competition and the C150 competition. Specifically, EDI requirements were formally incorporated into the selection criteria and institutional recruitment process (Government of Canada, 2016b, 2017). As a result, the third CERC competition (currently n = 8) and the C150 competition resulted in significantly increased diversity among chairholders, with over 50% of chairholders from each of these competitions self-identifying as women^{Footnote 25} and over 20% of C150 chairholders and Competition 3 CERC chairholders self-identifying as a member of a visible minority.

The significant increase in the diversity of chairholders that results when EDI is embedded in program requirements underscores the need to ensure EDI remains a strategic, sustainable and systemic consideration in both the CERC and C150 programs. Although the C150 competition technically attained the general labour force representation of 4%, there remains underrepresentation with respect to persons with disabilities. Again, specific proportions are not available given the small sample size so as to protect the privacy of chairholders.

There are a number of potential explanations for the low representation of persons with disabilities among CERC and C150 chairholders and the low representation of Indigenous Peoples among CERC chairholders. Key informants commonly made the observation that underrepresentation of CERC and C150 chairholders among these two designated groups is likely influenced by the small pool of researchers self-identifying as such, both in Canada and internationally. Indeed, across all disciplines, individuals self-identifying as Indigenous remain significantly underrepresented in academia, comprising only 1.4% of university professors in Canada (contrasted with almost 4% in the labour force) (Canadian Association of University Teachers; CAUT, 2018). Based on representation alone, this is an issue worth greater attention and discussion. Available evidence suggests a similar underrepresentation of persons with a disability in academia. According to the National Longitudinal Survey of Children and Youth (NLSCY), subsequently linked to tax information, individuals diagnosed with a physical and mental health condition in their youth were between 17 and 29 percentage points less likely to enroll in postsecondary education compared to a control group of individuals not diagnosed with one or more of these conditions (Arim & Frennette, 2019).

The possibility of underrepresentation aside, the available data may be underestimating the actual prevalence of researchers in these two designated groups. EDI data collection efforts in general are still in their infancy. Although Statistics Canada is currently developing improved reporting practices to more accurately capture breadth in the measurement of the four designated groups, existing data may actually be an underestimate of the actual level of diversity among academics (and other populations). Regardless of reporting refinements, individuals may simply be reluctant to self-report membership to these two specific designated groups for varied reasons—consider, for example, the stigma that might be associated with having an invisible disability (e.g., a mental health diagnosis) in a highly competitive environment that values high achievement and intellectual prowess (Canadian Mental Health Association, 2019), or cases involving ambiguity around one’s Indigenous identity (e.g., being Indigenous to one’s country of origin, but not Indigenous to Canada). In sum, it is plausible that institutional recruitment processes for CERC and C150 do not adequately capture the true extent of self-identification within these two groups.

3.3 CERC core team members

Opportunity to work with CERC chairholder is one of the primary reasons for faculty and HQP to join the CERC core teams

Faculty or highly qualified personnel (HQP: undergraduates, graduates, postdoctoral fellows, research technicians, research associates and other technical or research personnel) are recruited to work with the CERC chairholder as CERC core team members. As evidenced by the survey results presented in Figure 1, in addition to being afforded the opportunity to conduct innovative research and hone their skills, the opportunity to work with the chairholder and to have access to state-of-the-art infrastructure were the primary reasons for which faculty and HQP opted to join the CERC team. These findings were supported by interviews conducted with faculty and HQP included in the case studies, although both former and current team members more consistently indicated that it was primarily the reputation and calibre of the chairholder and interest in the work being undertaken by the CERC that were the key factors attracting them to the CERC core team.

Source: Survey of CERC core team members

Figure 1 long description

Figure 1: Factors that encouraged core team members to join the CERC core teams

Conducting innovative research, increasing skills and opportunity to work with the CERC encouraged faculty and HQP to join the CERC core team

Factor	Percentage	Frequency
Immigration regulations	17%	97
Personal reasons	37%	207
Collaboration with partners	65%	364
Location of institution	69%	387
Application of research findings	78%	437
State of research infrastructure	84%	473
Work with CERC	85%	477
Increase skills	92%	519
Conduct innovative research	92%	520

Source: Survey of CERC core team members

Composition of CERC core teams varies by member type (faculty and HQP) and priority area

Based on the five-year period between 2013 and 2017, the median CERC team consists of 29 core members in a given year (ranging between eight and 202 core team members per CERC). When examining faculty separately, only data between 2014 and 2017 were available, and show a median team count of four (range = 1 to 75). When considering HQP separately, the median team count is 27 personnel (range = 7 to 131). Upon further disaggregating the different groups of HQP (i.e., undergraduate students, master’s students, PhD students, postdoctoral fellows, and other HQP, respectively), results indicate a fairly even split between these groups on core teams, with a median number of approximately five personnel per HQP grouping.

Given the wide range of reported core team personnel in annual reports, core team size and distribution were also analyzed by research priority area (as aligned with federal government priorities) to determine if area of research might be a factor influencing the size of a research team. Table 3 illustrates that the priority area associated with the largest CERC teams is environmental sciences and technologies (EST), showing a median team count of 46 members, in comparison to the median team count of 23.5 members on a CERC team in a health-related area.

Table 3: Total median team size, split by priority area

Priority Area	n CERCs	Median number of faculty	Median number of HQP	Median number of total personnel
Environmental sciences and technologies	5	7.50	40.00	46.00
Natural resources and energy	5	9.00	28.00	31.50
Information and communications technologies	6	4.50	26.75	30.50
Health and related life sciences and technologies	8	3.00	20.50	23.50
Other (“open” priority areas; e.g., automotive industry)	2	4.50	13.75	18.25

Source: Statement of account annual reports 2013-17

---

Almost half of the core team members surveyed (46%; n = 257) indicated they were already at the CERC host institution upon joining the core team. Though not surprising, faculty (59%, n = 56) were more likely than HQP (43%, n = 199) to have already been based at the host institution before joining the CERC core team.^{Footnote 26 Footnote 27} Of those core team members who came from a different institution, just under a third came from another institution in Canada (29%; n = 72) and just under a third came from Europe (32%; n = 80). The remaining were primarily based in the United States (13%; n = 33), Asia (12%; n = 31) and South America (8%; n = 20). Similar proportions were generally evident when disaggregating faculty and HQP, with one exception: a higher proportion of faculty versus HQP came from an institution in the United States (29% vs. 11%).

3.4 Diversity of CERC core team members

According to survey data, approximately half of the CERC core team members self-identify as belonging to one or more of the four designated groups (45%, n = 252), as shown in Figure 2. Just over one-third of core team members self-identified as a woman (34%; n = 190), with a higher proportion of women represented among the HQP group (38%, n = 168) compared to the faculty group (20%, n = 19).^{Footnote 28} Just under one-fifth of core team members self-identified as a member of a visible minority (17%; n = 95), also with a greater proportion of HQP (20%; n = 85) than faculty (8%; n = 7) self-identifying as such.^{Footnote 29} Of the 17% of survey respondents that self-identified as a member of a visible minority, about a third (42%; n = 40) self-identified as Asian, followed by less than one-quarter who self-identified as Middle Eastern (16%; n = 15).

Similar to results related to CERC chairholders, a very small number of core team members self-identified as a person with disabilities (2%; n = 11) or as Indigenous (n < 5).^{Footnote 30} As discussed above, chairholders and institutional representatives express difficulty in recruiting researchers who self-identify within these two designated groups, perceiving that there is a smaller pool of eligible candidates. Beyond evidence of underrepresentation of these two designated groups within the academic environment both in Canada and internationally (e.g., CAUT, 2018; Mohamed & Beagan, 2019; Staniland, Harris, & Pringle, 2019), there is also the possibility that underrepresentation is at least partially an artefact of the data: individuals may choose to not self-identify for a multitude of reasons (e.g., stigma) and the actual number of individuals within groups may in fact be higher than is currently suggested. Responses from some of the chairholders and institutional representatives signal a need for further training within the academic environment to ensure that the systemic barriers in the research ecosystem, the benefits of EDI within research (on teams and within research design), and the connection between increased EDI and increased research excellence (e.g., Hewlett et al., 2013; SSHRC, 2019) are well understood.

*Respondents self-identifying with at least one designated group is 45%, n = 252.
*Total does not add to 100% as respondents were able to self-identify with more than one group.
*In keeping with the Privacy Act, numbers lower than five were suppressed to protect the privacy of respondents.

Source: Survey of CERC core team members

Figure 2 long description

Figure 2: Self-identification of CERC core team members

About half of CERC core team members identify as belonging to at least one of the four designated groups

Designated group	Percentage	Frequency
Prefer not to answer	6%	34
Indigenous Peoples	<5%*	n<5
Persons with disabilities	2%	11
Visible minorities	17%	95
Women	34%	190
Do not identify with any of these groups	49%	276

*Respondents self-identifying with at least one designated group is 45%, n = 252.
*Total does not add to 100% as respondents were able to self-identify with more than one group.
*In keeping with the Privacy Act, numbers lower than five were suppressed to protect the privacy of respondents.

Source: Survey of CERC core team members

---

4.0 Enhanced research capacity at Canadian universities in strategic areas of importance to Canada

To reiterate, given the infancy of C150, any outcomes discussed with respect to enhanced research capacity (and sustainability) as a result of funding is restricted to the CERC program. In the context of this evaluation, enhanced research capacity is examined through the quantity and impact of a CERC’s research outputs and knowledge mobilization activities while at the host institution, the degree to which CERC program has facilitated the forging of collaborations and partnerships, and the institutional growth that the CERC has catalyzed (i.e., increase in faculty, HQP, programs, facilities, and/or linkages with other institutions that would not have otherwise occurred).

4.1 CERC chairholder performance since award

Chairholders show significant increases in bibliometric indicators pre- to post-award

CERC chairholders demonstrated significant, positive increases in their publication productivity after the start of their award dates relative to their own performances over the 10-year pre-award period. This trend is also evident among matched Tier 1 CRCs, which, as outlined earlier, was deemed a valid comparison group given that both programs target established researchers who are acknowledged to be world leaders in their field. Specifically, both CERC and matched foreign and Canadian CRC1 chairholders significantly increased their average annual publication outputs from pre- to post-award, showing absolute increases of 22% (for CERCs), 28% (for foreign CRC1s) and 31% (for Canadian CRC1s; see Figure 3). Moreover, CERC and matched foreign CRC1 chairholders experienced significant increases in average annual international co-publication rates from pre- to post-award. Figure 4 displays the average rate at which CERCs and CRC1s publish research with international partners/collaborators, showing percent increases from pre- to post-award of 22% (for CERCs), 26% (for foreign CRC1s) and 4% (for Canadian CRC1s).

Source: Bibliometric analysis performed by Science-Metrix.

Figure 3 long description

Figure 3: Average annual publication output of CERCs and CRC1s pre- and post-award

CERCs and CRC1s show positive increases in publication output from pre- to post-award

Group	-	Average papers per year
Established CERCs	Pre-award	11.6
	Post-award	14.2
CERC-matched foreign CRC1s	Pre-award	8.6
	Post-award	11.0
CERC-matched Canadian CRC1s	Pre-award	7.4
	Post-award	9.7

Source: Bibliometric analysis performed by Science-Metrix.

---

Source: Bibliometric analysis performed by Science-Metrix.

Figure 4 long description

Figure 4: International co-publication rates of CERCs and CRC1s pre- and post-award

CERC and foreign CRC1s show positive increases in international co-publication rates from pre- to post-award

		Average papers per researcher per year	International co-publications rate
Established CERCs	Pre-award	11.6	44%
	Post-award	14.2	66%
CERC-matched foreign CRC1s	Pre-award	8.6	45%
	Post-award	11.0	71%
CERC-matched Canadian CRC1s	Pre-award	7.4	41%
	Post-award	9.7	46%

Source: Bibliometric analysis performed by Science-Metrix.

---

Bibliometric analyses were also conducted to determine the respective impact of the research produced by CERCs and CRC1s. Mirroring trends observed among Tier 1 CRCs, increases in HCP indicators for CERC chairholders post-award were modest rather than pronounced. In the instance of the ARC, a very slight decrease was observed among CERCs and foreign CRC1s (see Table 4). However, given the sensitivity of the CFREF to distributional outliers, some bibliometricians recommend greater reliance on the HCP as the latter is year- and field-normalized (Nature Index; Science-Metrix, personal communication, October 2018). In brief, the bibliometric indicators of CERC chairholders significantly exceed those of matched Tier 1 CRCs at each time point; with respect to the HCP10%, only CERCs showed an increase (albeit modest) from pre- to post-award.

Table 4: Key bibliometric indicators pre- and post-award for CERCs and CRC1s

Program	Pre-awardFootnote 1	Post-award	Difference
ARC
CERC	2.27	2.14	-0.13
CRC1 (foreign)	1.70	1.45	-0.25
CRC1 (Canadian)	1.73	1.82	0.09
HCP 10%
CERC	24.6%	27.5%	2.9%
CRC1 (foreign)	20.9%	18.8%	-2.1%
CRC1 (Canadian)	22.1%	20.9%	-1.2%
HCP 1%
CERC	3.6%	3.9%	0.3%
CRC1 (foreign)	2.2%	0.4%	-1.8%
CRC1 (Canadian)	1.9%	2.8%	0.9%
Footnotes Footnote 1 The pre-award figures in this table do not match those in Table 1 above because Table 1 presented results for a 10-year period pre-award and the results in this table present results pre-award for periods ranging between three and five years, depending on data availability for a given CERC and/or matched CRC1. Return to footnote 1 referrer

Source: Bibliometric analysis performed by Science-Metrix.

---

There are several potential explanations for the modest rather than pronounced increase in publication impact. The first is a possible ceiling effect—that is, in the pre-award period, CERC chairholders were already world-class researchers with high levels of funding. Additionally, as evidenced in the CERC evaluation conducted in 2013-14, there is typically a “recovery period” post-award whereby the logistics involved in building infrastructure and establishing a new team can take time away from research production. Relocating a full research program from one institution to another is a demanding endeavour, particularly when it involves moving to a new country. This transition requires, at the very least, building a new pool of undergraduate, master’s and doctoral students, as well as postdoctoral fellows; cementing relationships with potential new partners and collaborators; acquiring and calibrating instruments; and navigating a new set of administrative practices. The post-award figures presented in Table 4 encompass this recovery period, thus dampening the overall effect of post-award productivity. It is to be expected then that performance increases over the baseline would only be achieved after a longer time frame. As also evident in Table 4, a smoother transition is observed among matched Tier 1 Canadian CRCs who are not faced with as many logistical obstacles as CERCs and foreign CRCs in the immediate post-award period.

Another potential explanation for only modest increases in CERC publication impact as measured through bibliometric indicators is that, with their influx of funding, CERCs may be compelled to conduct overall riskier research that, in turn, might take more time to develop and cannot yet be captured by bibliometric analysis. A few faculty members participating in the CERC case studies confirmed that, indeed, the freedom and flexibility of CERC funding allowed them to pursue more challenging and higher-stakes research ventures than would have been possible otherwise. It has been found that riskier research typically takes more time both to be published and to be cited, although these initial delays are often contrasted by very strong citation profiles in the long term (Wang, Veugelers, & Stephan, 2015). Given that research projects can take years to complete, chairholders will likely experience (or continue to experience) an increase in CERC-related research outputs and outcomes following the completion of their terms.

It is important to bear in mind that publication volume is but one of many indices of research capacity and contributions. Moreover, as articulated in the limitations section of this document, we were only able to quantify the publication output of the CERC chairholder—we were not able to track the activity of the entire team. As such, the publication metrics reported here are likely underestimates of the actual increase in publication rates associated with CERC funding.

CERC chairholders have a positive impact on the productivity of host institutions

Host institutions produced a significantly greater number of annual publications in fields associated with the CERC relative to comparison groups. More specifically, host institutions experienced increases in publication output of roughly 10 articles per year above the trends of Canadian and international comparators, representing a 13.3% increase (see Figure 5). Importantly, this lead vanished if publications authored by CERC chairholders were removed from the institutional profiles. It was found, however, that these leads were driven by exceptional performances from a few CERC teams rather than by gains evenly distributed across the population of chairholders.

Source: Bibliometric analysis performed by Science-Metrix.

Figure 5 long description

Figure 5: Average number of annual publications per year of host institutions with and without CERC articles and matched Canadian and foreign institutions

Host institutions saw a significantly greater number of annual publications in fields associated with the CERC relative to comparators

	Average papers per year
Host institutions	92.1
Host without CERC papers	81.3
Matched Canadian institutions	82.8
Matched foreign institutions	81

Source: Bibliometric analysis performed by Science-Metrix.

---

4.2 Institutional growth

One of the expectations of the CERC programs is that it will provide host institutions with the ability to grow, with institutional growth defined as an increase in the number of faculty, HQP, programs, facilities, and/or linkages with other institutions that would not otherwise have occurred had the CERC not been hosted at their institution. Overall, 96% of host institutions reported that CERCs were instrumental in stimulating institutional growth over the evaluation period. Interview and case study findings indicate that the CERCs have fostered institutional growth in a number of different ways, including by attracting a number of high-calibre faculty and HQP, enhancing existing research centres, building new infrastructure in CERC-supported research areas, supporting the establishment of research institutes and groups, allowing for leveraging of funding or support from other sources and helping to develop the international reputation of the host institution.

Regarding building new infrastructure in CERC-supported research areas, in total, CERC administrative data show that a median of $1 million of CERC funds were used to support infrastructure needs during the period covered by this evaluation. In addition, CERCs collectively received $12 million from CFI over this evaluation period (and a total of almost $24 million since the launch of the CERC program). Of note, there is wide variability in infrastructure-based spending of CERC funds across priority areas. The natural resources and energy priority area, for example, is associated with the highest infrastructure cost, with a minimum reported cost of nearly $550,000 in a given year (Mdn = $1.4 million). The second highest infrastructure cost is associated with the health priority area (Mdn = $1.3 million).

4.3 Collaborations and partnerships^{Footnote 32}

The vast majority (over 80%) of chairholders agree that the CERC program, through its scale and calibre, had a great influence on their ability to establish partnerships and collaborations. Overall, 92% of CERCs reported engaging in partnerships and 82% reported engaging in international partnerships over the four-year period from 2014 to 2017, with a median of three partnerships reported in a given year. Overall, 96% of CERCs have reported engaging in both national and international collaborations over the same four-year period, with a median of four ongoing collaborations in a given year.^{Footnote 33}

Case studies and key informant interviews underscored that their funding, reputation as chairholders and the quality of the research produced by their strong core team resulted in increased requests for media appearances, in addition to an influx of requests to deliver public lectures, attend international meetings, and explore professional relationships with groups and individuals they would not have otherwise had the opportunity to engage with. Moreover, access to financial and infrastructure resources through the CERC program allowed chairholders to leverage additional funding. According to administrative data, CERCs report a median amount of just under $1 million per year in external funding to support research endeavours, primarily from public organizations. Despite evidence of cross-sectoral collaborations and partnerships, Figure 6 clearly illustrates that CERCs partner and collaborate primarily with academic institutions; in particular, academic collaborators are almost four times more prevalent than collaborators from any other sector. This is not altogether surprising given the academic environment within which these teams operate.

In a similar vein, the CERC program provides students and other HQP with opportunities to attend conferences, develop national and international partnerships and collaborations, and build their own professional networks. Almost two-thirds (62%; n = 350) of HQP survey respondents indicated that being part of the CERC core team had a positive impact on their ability to establish national and international relationships to a great or very great extent—a sentiment that was articulated repeatedly in interviews with current and former HQP participating in case studies. Indeed, case study interviews with HQP noted that the types of collaborations and networks they have been able to establish would not have been possible at their current career stage without their membership on the CERC core team.

Source: Chairholder annual reports 2014-17

Figure 6 long description

Figure 6: Median number of CERC partnerships and collaborations by sector covering the period 2014 to 2017

CERCs report a greater median number of partnerships and collaborations with academic organizations

	Median number of partnerships	Median number of collaborations
Academic institutions	4	26
Public organizations	3	7
Private organizations	3	5
Not-for-profit organizations	2	3
Other organizations	1	4

Source: Chairholder annual reports 2014-17

---

4.4 Research outputs

CERCs are producing high levels of research output

Based on self-report data from annual reports, over 2016 and 2017, CERCs collectively produced a total of 2,336 research outputs, with a median value of 52.5 outputs per CERC. The most common research outputs in both 2016 and 2017 were contributions to scholarly refereed journals, with a median output of 17 per CERC in a given year, followed by conference proceedings with a median output of 10 per CERC in a given year. Scholarly refereed journal articles and conference proceedings account for 56% and 26% of all reported research outputs, respectively, whereas public policy and government publications account for less than 1% each (see Figure 7). The fact that CERC outputs tend to centre on peer-reviewed publications and conference proceedings is not surprising given that all CERCs are based at academic institutions and are expected to conduct research tailored to academic audiences in order to sustain their funding portfolios.

As an example of the impressive productivity of the CERCs, bibliometric data revealed that one of the chairholders alone produces 10% of the world’s publications in their area of research, and is responsible for approximately 12% of all the citations in their field. Moreover, this particular CERC team includes five out of the 10 top-cited researchers in their field.

Source: Chairholder Annual Reports (2016, 2017)

Figure 7 long description

Figure 7: Distribution of research output by type for 2016-17

Most CERCs' research output is through scholarly refereed journals

	Percentage	Frequency
Scholarly refereed journals	56%	1302
Conference proceedings	26%	609
Books/chapters	4%	96
Research reports	4%	93
Patents	1%	29
Copyright	1%	17
Article in professional/trade journals	Less than 1%	10
Public policy	Less than 1%	9
Government publications	Less than 1%	3
Other	7%	168

Source: Chairholder Annual Reports (2016, 2017)

---

By research area and type of research output, environmental sciences and technologies-related (EST) CERCs produce almost half (47%) and the largest proportion of all research outputs compared to other priority areas (health, information and communications technologies, natural resources and energy, and other), which is reasonable given the larger median team size associated with EST (see Table 3). Copyright research outputs have exclusively been reported by CERCs in the health priority area, who also produce 60% of research outputs appearing in professional/trade journals. Also notable is the fact that almost half of patents (45%) were filed by CERCs in the information and communications technologies priority area.

According to the survey results, while on the CERC core team, the majority of HQP produced conference publications and/or presentations (66%; n = 262) and peer-reviewed journal articles (57%; n = 228), which aligns with the findings of the overall CERC teams presented in Figure 7. A third of HQP (33%; n = 132) also indicated producing reports, briefs, and other forms of grey literature, and one in five produced tools for research-related activities (e.g., websites, audio-visual products, databases).

CERCs are transferring knowledge mainly to international and private organizations

As shown in Figure 8, CERCs most frequently provide expert advice to international (29% of reported knowledge transfer activities) and private (21% of reported knowledge transfer activities) organizations, accounting for 278 out of 556 total activities reported by CERCs over the last two years.

Source: Chairholder Annual Reports (2016, 2017)

Figure 8 long description

Figure 8: Distribution of expert advice by type for 2016-17

CERCs tend to provide expert advice to international organizations

	Percentage	Frequency
International organizations	29%	161
Private organizations	21%	117
Not-for-profit organizations	15%	85
Other Canadian government officials	49%	50
Other international government officials	8%	45
Elected Canadian government officials	7%	40
Elected international government officials	1%	7
Other	9%	51

Source: Chairholder Annual Reports (2016, 2017)

---

Similar to the low prevalence of research outputs from academics in government (<1%) and public policy contexts (<1%) highlighted in Figure 7, interviews with CERC chairholders revealed that only one-fifth of them (n = 3) reported having the opportunity to provide advice to public sector organizations. As explained by one chairholder, opportunities for academics to provide advice directly to government seemed limited in comparison to what they were used to abroad. In their view, this was one of the shortcomings of the CERC program, as they felt that chairholders had been underutilized as a resource that could contribute to policy and decision-making in Canada. Indeed, the CERC logic model specifies a long-term outcome (seven to 10 years post-award) in which “individuals and organizations from outside of the academic sector benefit from the research conducted by CERCs” (see Appendix A). It may be the case that coming from outside of Canada, chairholders are not well-versed or well-connected with Canadian public policy contexts, precluding them from making more significant contributions in this area.  Alternatively, chairholders may not always be directly aware of when and how their research is being applied to public policy contexts, in which case the cited number in Figure 7 may be an underestimate.

Notwithstanding this concern, almost all first cohort chairholders could provide examples of discoveries and insights that could have applied implications for government, industry, and medicine. For example, chairholders highlighted that their discoveries and insights have led to the development of technologies and applications of importance, the generation of new knowledge or understanding of phenomena and, more indirectly in terms of implications for government, industry, and medicine, published their work in high-impact journals. Although second cohort chairholders are just beginning to see the impacts of their research, evidence from Competition 1 CERCs strongly suggests that their efforts will have a wide impact in the near future.

---

5.0 Sustained research capacity at Canadian universities in strategic areas

In addition to the enhancement of research capacity stimulated by CERC teams (refer to Section 4.0), the sustainability of research capacity is also an important goal of the program. In the context of this evaluation, sustainability is operationalized by the growth and retention of core team members, partnerships and collaborations built through the CERC, and the continued production of a high volume of impactful research outputs. In essence, the CERC research program continues to enhance the host institution’s international reputation while taking advantage of the momentum built during the CERC term (Tri-agency Institutional Programs Secretariat, 2017a).

Given that Competition 1 chairholders have only recently completed their terms, the next evaluation of the CERC program will be more conducive to determining the extent to which the program has allowed for the creation of research capacity that is sustainable beyond the end of the CERC term. However, it is still possible to gauge sustainability in a preliminary way in this evaluation by examining chairholders’ and core team members’ intent to remain at their host institution and/or in Canada, as well as the perceived impact of the end of CERC funding on partnerships and collaborations.

5.1 CERC chairholders remaining at host institution

At the time of data collection, Competition 1 chairholders had either completed or were nearing completion of their seven-year terms. The vast majority of these Competition 1 chairholders (78%; n = 14) planned to remain at their host institution after their CERC terms.^{Footnote 34} Case study and interview data suggest that any uncertainty about remaining at the host institution was strongly linked to the chairholder’s ability to secure future funding and to the host institution’s commitment to sustaining the research program—two factors that are still uncertain at the time of this evaluation. Although the majority (64%; n = 25) of chairholders and institutional representatives reported that the CERC research program would be impacted by their ability to maintain a level of funding equivalent to the CERC award, the chairholders who plan to remain at their host institution maintain that the success and strengths of their research program, the infrastructure that they have built, and the collaborations they have established are too significant to leave behind. In many cases, external funding sources like CFREF will allow them to sustain a good portion of the CERC research program and team in a way that would not have been possible without the additional funding. As indicated earlier, about 75% of CERC chairholders from the first two competitions are directly involved in CFREF initiatives; four of these CFREF initiatives are being led by CERC chairholders. In this sense, synergies existing between tri-agency programs increase the sustainability potential of CERC-led research programs. Moreover, a few chairholders (22%; n = 2 of the 9 case study chairholders) noted that the institutional support that they receive (financial and otherwise) is a key reason for deciding to stay at the host institution.

5.2 Retention of CERC core team members

CERC core teams have grown over time

CERC core teams have grown overall between 2013 and 2017 (see Figure 9), indicating that institutions are not only retaining researchers during this time period, but also attracting more personnel. Interestingly, when examining HQP and faculty separately, trends diverge, showing a growth in HQP over the five-year period (primarily driven by students rather than other forms of HQP) but a decline in faculty personnel over that same period. The decline in faculty suggests that institutions and CERCs are initially attracting faculty to CERC core teams, but may have some difficulty retaining these faculty members over the longer term, potentially having a negative implication on sustaining long-term capacity.

Source: Chairholder annual reports, 2013-17

Figure 9 long description

Figure 9: Growth trends of CERC core teams from 2013-17

Overall, core teams have grown in the last 5 years

	2013	2017
HQP	26	37
Faculty	6	4
Total	26	37

Source: Chairholder annual reports, 2013-17

---

On average, CERC core team survey respondents indicated that they were part of their CERC core teams for about three years, ranging from a minimum of two months to a maximum of just over nine years. As would be expected, HQP, being more transient and at an earlier stage in their careers, were more likely to be a part of the CERC core team for a shorter time period than faculty (three years vs. four years).

CERC core team members provide varied reasons for leaving the CERC

The overall growth of CERC core team size (as indicated in Figure 9) does not necessarily imply that retention of CERC core team members is high. In fact, over one-third of CERC core team survey respondents (34.7%; n = 193) indicated they are no longer part of the CERC core team. Not surprisingly, a higher proportion of HQP (38.2%; n = 176) indicated they left the CERC core team compared to faculty (17.7%; n = 17),^{Footnote 35} given that HQP include students and postdoctoral fellows who are likely to move to another institution or find employment elsewhere once their studies or contract end. This expected transience was confirmed by the fact that the most common reasons survey respondents left their CERC core teams included finding employment elsewhere (33%; n = 64), their contract coming to an end (33%; n = 64) and having completed their studies (26%; n = 50). The reasons provided by faculty and HQP for leaving the CERC team were similarly varied, with the notable exception of 18% (n = 3) of faculty who retired from their positions.

Few survey respondents cited negative reasons for leaving the CERC core team (less than 10%). The most common negative reasons for leaving included overall dissatisfaction with the program, leadership, and opportunities (5%; n = 6); and the chair leaving their position (4%; n = 5).^{Footnote 36} Overall, the CERC core team survey results and interviews with faculty and HQP revealed that the vast majority of CERC core team members enjoyed their experience on their respective teams.

As shown in Figure 10, of the survey respondents who are no longer part of the CERC core team and are employed full- or part-time (n = 129), almost half are employed outside of Canada (45%; n = 58) while the other half are employed in Canada either at the CERC host institution (21%; n = 27) or elsewhere in Canada (34%; n = 44). Notably, of the CERC core team member survey respondents who came from abroad to join the CERC core team, about a quarter (23%; n = 11) remained in Canada after having left the team.

Source: Survey of CERC core team members

Figure 10 long description

Figure 10: Employment location of CERC core team members no longer a part of the CERC

The majority of faculty and HQP no longer part of the CERC remain employed in Canada

	Percentage	Frequency
Do not know/prefer not to answer	1%	1
Employed in Canada - same institution as CERC	21%	27
Employed in Canada - not at CERC institution	34%	44
Employed outside Canada	45%	58

Source: Survey of CERC core team members

---

Of those survey respondents who indicated they were former CERC core team members and had secured employment (in Canada or abroad) that was either full- or part-time (refer to Figure 10), about half (49%, n = 64) indicated that they continue to work in the university sector. Another third (29%; n = 38) are working in the private sector and 14% (n = 18) are working in government. The case studies provided supporting evidence to the survey data that former HQP are being hired into faculty positions at their host institutions or at other universities in Canada and abroad, as well as in industrial positions in Canada and abroad. For those who remain in Canada, there is still a potential benefit to the country in terms of sustainable gains from their CERC training experiences. The fact that half of former CERC core team members reported securing employment in academia is impressive when contrasted with the general trends among PhD graduates in Canada. A recent study examined employment outcomes of over 10,000 students graduating from a PhD program at the University of Toronto between 2000 and 2015 (Reithmeier et al., 2019). Findings indicate that only 18% of those earning a PhD in the life sciences and 22% of those earning a PhD in the physical sciences subsequently pursued a career in the university sector.

Interestingly, former HQP from over 90% of case studies indicated that they were hired in the fields targeted by their CERC, whereas only about half (52%; n = 67) of the survey respondents indicated the same. The discrepancy between the survey and case study findings could be attributed to the greater likelihood of CERC chairholders identifying former core team members with whom they are still affiliated or former core team members currently working in similar fields,^{Footnote 37} resulting in a lower representation of case study interviewees currently working outside of the CERC research field.

Most CERC core team members plan to still be in Canada in five years

Survey respondents who specified that they were still part of their CERC core teams (59%; n = 330) were also asked to estimate what their work situation would likely be five years from now. Almost half of respondents (41%; n = 135) believe they will still be working at the CERC institution; one-quarter (25%; n = 82) expect to be working at a different Canadian institution or elsewhere in Canada. Only a small proportion of respondents expect to be working abroad at another institution or organization (9%; n = 30) and a similarly small proportion expect to return to their previous institution (2%; n = 6).^{Footnote 38}

As would be expected, faculty (79%; n = 49) were more likely than HQP (40%; n = 83) to indicate they would still be working at the CERC institution in five years, consistent with the idea that faculty are relatively more rooted in their careers compared to HQP.^{Footnote 39 Footnote 40} Furthermore, HQP (37%; n = 76) were more likely than faculty (9%; n = 6) to indicate they would be working at another institution in Canada or be employed elsewhere in Canada, again consistent with the notion that HQP are likely to move between institutions and organizations as they progress through academia.^{Footnote 41}

Common reasons for which respondents anticipated they would be studying or employed outside of Canada in five years include the fact that their family and/or social network are primarily elsewhere (37%; n = 11), they are more attracted to research opportunities elsewhere (23%; n = 7) or they have already secured employment elsewhere (13%; n = 4).

5.3 Retention of collaborations and partnerships built through the CERC program

Given the important role that the program played in establishing partnerships and collaborations, about half of chairholders (46%; n = 11) expressed a moderate to extreme level of concern about their ability to maintain these after CERC funding ends. However, while chairholders agreed that CERC funds were important for establishing partnerships and collaborations, most were confident that these could be maintained conditional on their ability to continue their research program with alternative sources and similar levels of funding (recall that collaborations and partnerships established by CERCs provide annual external funding of nearly $1 million to support CERC research initiatives). Despite some concern around the sustainability of these partnerships and collaborations after CERC funding ends, one first cohort chairholder was relatively optimistic, expressing that because their CERC program is stable, a certain level of trust has been established between their research group and partners. In turn, this built trust bodes well for a continued prosperous relationship with these entities. Among those predicting a moderate impact of the end of CERC funding on collaborations, one second cohort chairholder anticipated that their partnerships would continue, but perhaps not on the same scale.

---

6.0 Design, delivery and efficiency

6.1 Overall design and delivery

Overall, key informant and case study findings suggest that although some design features of the CERC and C150 programs have been successful, other features present certain challenges for institutions, chairholders and core teams.

Key strengths of the CERC program

Key features of the CERC program that were highlighted as strengths by key informants and case study participants included the flexibility and latitude of the program and its funding, synergies with other federal funding programs (e.g., CFREF), the value and prestige of the award, its ability to attract top-tier international researchers and build a program of research from the ground up, and the selection of CERCs in strategic areas of research for Canada. In addition to allowing faculty and HQP to work with the world-class chairholder, case study findings underscored additional factors that attracted core team members:

• access to state-the-art equipment and infrastructure
• opportunities for networking (collaborations, partnerships)
• opportunities to build new skills
• being viewed as more marketable and reputable as a result of their membership on the CERC team
• the importance and impact of the research
• support for more innovative, “riskier” research

In sum, a strong majority of stakeholders—namely institutional representatives—shared positive feedback about the program, commonly expressing that the program was necessary to attract the calibre of researcher in question.

Key challenges of the CERC program

Application process: The timeframe for the application and nomination processes for the CERC competition was considered too lengthy and onerous by a few chairholders and some vice-presidents of research, ultimately leading to the loss of desirable candidates who accepted other job opportunities. To date, there has typically been a three-year lag between the time a competition has been announced and the start date of many chairholders within that competition. Interestingly, some noted that the specific timeframe associated with Phase 2 of the competition (i.e., for the chairholder to officially decide whether to accept the award and to relocate) was too short given the magnitude of the decision. In addition, institutional representatives commonly mentioned as a shortcoming the resulting loss of the CERC opportunity for institutions whose candidates ultimately declined the chair to accept other offers.

Annual reporting: A few chairholders and institutions indicated that the annual reporting requirements were fairly onerous and lengthy, and at times impeded their ability to focus on scientific production. There was also the perception that annual report data were not always reviewed in a timely manner.

Term of the award: A common perception among chairholders and vice-presidents of research is that the seven-year term of the CERC award is insufficient (in terms of time, not award value) and that the influx of CERC funding ceases abruptly at the end of the term. Given the high value of the award, the end of the funding period is felt very strongly—particularly if chairholders have not yet secured commensurate funding from other sources. Delays in setting up research labs and other infrastructure (especially building or modifying a lab and purchasing equipment) at the beginning of the CERC terms was cited as a challenge, especially when considering the high level of progress that is expected within the program. Chairholders also noted that research needs and timelines vary according to research area, which could be aided by a term extension or “tapering-off” period. It should be noted that CERC chairholders are granted an automatic one-year extension as a phase-out period to expend their remaining funds; although the Terms and Conditions allow for the possibility of an additional extension with proper justification, this option was not universally understood among chairholders.

Misalignment of expectations: A few key informants (i.e., institutional representatives, chairholders, former HQP) and case studies noted that there can sometimes be a misalignment in expectations between the host institutions and the CERC chairholders, with a certain inability to ensure accountability. In some cases, institutional proposals and plans for the chair are not articulated in sufficiently concrete terms to ensure a congruent vision. Anecdotally, one institutional representative noted that the CERC program was not implemented as effectively as it should have been by the institution, which resulted in the chairholder’s expectations being unmet. Additionally, a chairholder from another case study indicated that a limiting factor to the success of their research program was the lack of autonomy accorded to the chairholder by the institution, an unexpected situation that created an abundance of additional challenges for the chairholder during their CERC term.

Securing external funding: Chairholders from a few case studies noted that they faced challenges in securing additional funding for their labs. Specifically, while some noted that there were delays in receiving funding from CFI, others shared the perception that the high value of the award seemed to impede their ability to secure other sources of tri-agency funding.

Lack of program visibility: A few key informants, coupled with observational data collected by evaluators during the data collection process, indicated a lack of visibility of the CERC program. It was clear during the data collection process that people working within the academic environment without specific linkages to the CERC program were generally unaware of the program’s existence and/or routinely confused it with the CRC program. Furthermore, during key informant interviews, a few institutional representatives and a CERC Competition 2 chairholder expressed that, indeed, they were surprised at the low visibility of the program and suggested that the CERC program could benefit from increased visibility.

Sustainability: Sustainability of the CERC-funded research program once the funding period ends was a commonly raised concern by chairholders and institutional representatives alike, specifically that the award is not renewable and that funding ends relatively abruptly at the end the CERC term. The level of concern around the perceived sustainability of the research program was in part mitigated by whether a chairholder had been able to secure new funding (such as through CFREF); accordingly, the level of concern is not consistent across CERC teams. Not surprisingly, the areas in which the end of the award is most commonly anticipated to have an extreme or large effect among chairholders and institutional representatives is in the level of funding (49%; n = 19), maintenance of research programs and projects (33%, n = 13) and retention of HQP (28%; n = 11). As noted in Section 5.3, maintaining collaborations and partnerships established through the CERC-funded research program once funding has ended is also an area of concern for chairholders.

Unintended impact: A minority of institutional representatives expressed that such large funds allocated to a single researcher and associated research team has caused some tension within their department (particularly among faculty) as a result of perceived inequities. A few institutional representatives even suggested that funds would be best distributed to multiple researchers so as to reduce or eliminate friction among faculty.

Key strengths of the C150 program

Some C150 design elements underscored as strengths by key informants (C150 chairholders, C150 candidates who declined, and institutional representatives) included the expedient nomination and application process, the openness of funding to multiple disciplines, the flexibility of funding allocation, and the impact of the award in raising the international profile of the institution.

C150 chairholders also expressed their excitement about the opportunity to work in Canada specifically, stating that Canada:

• values research education and training;
• values collaboration and engagement with the world;
• offers the freedom to pursue research that is intrinsically interesting;
• strongly supports inter-disciplinary research.^{Footnote 42}

Key challenges of the C150 program

Given the recency of the program, newly awarded C150 chairs and institutional representatives were only able to speak to challenges related to the application process.

Application process: A few key informants noted that the application process was too compressed and did not afford sufficient time to make such a large decision that would involve uprooting both their work and their families; the compressed competition timeline ultimately hindered the ability of institutions to recruit certain desired candidates. Across all interviewees (both chairholders and institutional representatives), the fact that the application process took place over the summer was a significant challenge, given the fact that many individuals were not readily available during this period.^{Footnote 43}

6.1.1 EDI-specific elements of design and delivery

CERC has had a history of EDI challenges

The CERC program has faced EDI challenges since the first competition launched in 2008, both in relation to chairholder and core team composition. Competition 1 resulted in the complete absence of women among the nominated chairholders. This absence of gender diversity raised significant concerns by program management and resulted in the creation of the Ad Hoc Panel on CERC Gender Issues in 2010 (Dowdeswell, Fortier, & Samarasekera, 2010).

EDI advances in the recruitment of chairholders

EDI considerations were formally embedded in the selection process of the third CERC competition. In addition to institutions reporting on their recruitment and outreach efforts through an EDI plan reviewed by the selection panel, individuals with EDI competencies were invited to be part of the peer review panels. The CERC program supports institutions in their efforts and provides documentation on recruitment best practices (Government of Canada, 2016b).

Similar EDI best practice guidelines were applied to the C150 program and to the current CERC competition (Competition 3), whereby recruitment and outreach processes were expected to be open, transparent, equitable and diverse in the context of this competition (Government of Canada, 2017). Notably, institutional recruitment and outreach processes were expected to place emphasis on the advertisement and the workings of the search committee. The advertisement for the chairholder position was to feature a strong commitment statement to EDI and encourage applications from members of the four designated groups. Keeping in mind that candidates’ record of research achievement is a recruitment criterion, the advertisement was to clearly acknowledge that career interruptions (due to illness, pregnancy, and the like) could impact researchers’ productivity and encourage applicants to explain this impact if relevant to their situation. Moreover, the search committee itself was to be composed of diverse members. The committee was also expected to receive training on unconscious bias and its consequences on the careers of members of the four designated groups. Finally, the committee was to integrate mechanisms that take into account career interruptions in the evaluation process, so that applicants would not be unfairly disadvantaged (Government of Canada, 2017).

Given the above measures, it is clear that substantial attention has been paid to the areas of EDI in the context of the most recent CERC and C150 competitions. As noted in Section 3.2, the third CERC competition and the sole C150 competition each resulted in over 50% of the chairholders being women. The C150 competition also resulted in almost a quarter of the chairholders identifying as members of a visible minority.

Remaining EDI challenges in the recruitment of chairholders

Although key informants generally agreed that there have been EDI-related improvements since the inception of the CERC program, some key informants perceived that these measures had not led to any significant progress. For example, panel members believed that the inclusion of an individual with EDI competencies on the review panels,^{Footnote 44} although a good idea in theory, was not particularly effective in that this person’s input was not well integrated in the decision-making processes.

Additionally, interview findings for both CERC and C150 programs strongly suggest that there is a perceived challenge at the institutional level in attempting to fulfill EDI considerations because the recruitment of researchers of the calibre sought for these programs limits the pool of potential applicants. The general recruitment of individuals of the calibre of researchers sought for chairholders was noted as a challenge in itself (not only among traditionally underrepresented groups) as the pool of researchers of this calibre within their field of study was perceived as already limited. This points to the need to underscore the importance that institutions use a broad and flexible definition of research excellence, not looking only at publication data and traditional research impact. Lastly, although there has been increased emphasis on EDI considerations for both CERC and C150 programs, there was a general lack of clarity and understanding about EDI requirements and guidelines across institutional representatives (e.g., expectations surrounding content of EDI plan and strategy), with some vice presidents of research indicating they had to revise the EDI strategy portion of their application multiple times before it was accepted by the review panel. These findings signal a need to clarify requirements and further develop EDI competencies.

EDI advances in the recruitment of core team members

The third CERC and sole C150 competitions introduced EDI measures that specifically focused on the core team. In their applications, institutions were required to include a detailed equity plan that considered EDI within the team composition (Government of Canada, 2016b). The CERC chairholder and institutional annual report templates were revised to include questions on EDI, and C150 chairholders must now provide data on the composition of their teams (Government of Canada, 2016b; Tri-agency Institutional Programs Secretariat, 2017a; Government of Canada, 2017; Tri-agency Institutional Programs Secretariat, 2017c). Evidence from key informant interviews suggests that most institutions have some form of EDI policy or strategy that helps guide chairholders with the recruitment of their research team. However, interviews with C150 chairholders themselves generally suggested that, at least at the time of data collection (shortly after the start of their terms), they themselves had not put a great deal of thought and energy into their EDI plan for recruiting a research team and tended to attribute this responsibility to the institution.

Interview findings suggest that, despite challenges, research teams generally include the participation of women and members of visible minorities. This is consistent with the findings from the CERC core team survey noted in Section 3.4 that about half of the respondents self-identify with at least one of the four designated groups.

Remaining EDI challenges in the recruitment of core team members^{Footnote 45}

Despite the increased representation of women and members of visible minorities among core team members, key informants and case study participants communicated their view that the pool of qualified candidates for core team members—in particular, candidates self-identifying as Indigenous, persons with disabilities, and (to a lesser degree) women and members of a visible minority—remains limited. This is particularly true in science, technology, engineering, and mathematics (STEM) research areas (CAUT, 2018). Multiple key informants expressed their perception that caution should be exercised in promoting diversity among team members at the expense of quality. Although most key informants did not go so far as to claim that there would necessarily be a trade-off between diversity and quality, they did imply that one might have to come at the expense of the other (given the perception that the pool of candidates was limited). The fact that this notion was expressed may partially reflect an unconscious bias at the institutional level surrounding recruitment and may speak to a need for further training on the systemic barriers that exist in the research ecosystem, the importance of employing a broader definition of research excellence that incorporates a diversity of perspectives, and the need to apply best practices around the recruitment of individuals from the four designated groups.

Only about one-third of surveyed core team members (31%; n = 172) indicated that the CERC or institution implemented adequate EDI-related measures to a great or very great extent (e.g., gender balance efforts, encouragement of cultural diversity). This relatively low proportion further suggests a need for future efforts and training in EDI at the institutional level.

Furthermore, a few core team members surveyed indicated that they had faced specific barriers to at least a slight extent related to their gender identity (5%, n = 26), or their self-identification as a member of a visible minority (2%, n = 10) or as a person with a disability (1%, n = 6). However, proportions increase when restricted to survey respondents who self-identified as belonging to a particular designated group: for example, 11% of respondents who self-identified as a member of a visible minority and 55% of respondents who self-identified as a person with a disability indicated that they had faced systemic barriers. Some specific examples include education/career interruption due to pregnancy or other factors, language barriers, and an arduous/lengthy immigration system.

6.2 Cost-efficiency

Consistent with the Treasury Board Secretariat’s reporting requirements, a cost-efficiency analysis was conducted for CERC to determine the extent to which the program was delivered in an efficient manner based on administrative expenditures in relation to grant expenditures.^{Footnote 46} The councils have most commonly assessed operational efficiency by calculating the ratio between a program’s operating costs and grant funds awarded within a given fiscal year. Data were provided by the NSERC-SSHRC Finance Division and TIPS.

CERC program

CERC program efficiency ratios from fiscal year 2013-14 to 2017-18 are presented in Table 5. Over this reporting period, the program cost between 4.54¢ and 7.72¢ to administer for every $1 granted, with an average of 5.49¢. Accordingly, operating expenditures for CERC ranged from 4.3% to 7.0% of total program expenditures (average of 5.20% over the reporting period). The program has experienced a fairly steady state over the last five years, with a slight increase in the operating ratio for 2017-18, likely related to the allocation of new chairs from Competition 3 and changes to the selection process (i.e., new EDI requirements).

For comparison purposes, SSHRC programs as a whole cost between 3.96¢ and 4.26¢ to administer for every $1 granted, and TIPS programs as a whole (excluding Research Support Fund; RSF) cost between 3.65¢ and 5.42¢ to administer for every $1 granted. Although CERC has a higher cost-efficiency ratio than the averages across SSHRC and TIPS programs, the administrative ratios associated with CERC are commensurate with other large Tri-agency funding programs such as the College and Community Innovation Program (CCI) and the Centres of Excellence for Commercialization Research (CECR), for which administrative costs represent between 5.00¢ and 6.00¢ for every $1 granted.

The cost-efficiency analysis for C150 was not included given the infancy of the program. It will only be possible to gauge the relative cost-efficiency and value for money of CERC versus C150 in the context of the next evaluation when research outputs and administrative costs will be available for the latter.

Table 5: CERC Program Expenditures and Efficiency Ratios

Fiscal Year	Total Grant Expenditures	Total Operating Expenditures	Operating Ratio (¢:$1) to Grant Funds Awarded	% Operating Expenditures / (Grant Expenditures + Operating Expenditures)
2013-14	$18,200,000	$1,123,999	¢6.18	5.82%
2014-15	$30,750,000	$1,633,940	¢5.31	5.05%
2015-16	$34,850,000	$1,649,347	¢ 4.73	4.52%
2016-17	$36,498,000	$1,655,759	¢4.54	4.34%
2017-18	$24,150,000	$1,864,528	¢7.72	7.17%
Total	$144,448,000	$7,927,573	¢5.49	5.20%

Source: CERC costing results from 2013-14 to 2017-18

---

7.0 Key findings and recommendations

Evaluation Question 1: To what extent do the CERC and C150 programs continue to address a unique need?

Key findings suggest that CERC and C150, namely due to their prestige and value, are unique in their ability to attract and support world-class international researchers in building research capacity within Canada. These programs represent a specific niche in federal funding programs. Despite a few institutional representatives expressing concern about the tension created among faculty as a result of the disproportionately high level of funding issued to a single research team, most stakeholders shared positive feedback about CERC: they commonly expressed that the program was necessary to attract the calibre of researcher in question. Overall, the perception among stakeholders, combined with the influx of world-class researchers into Canada and their noted productivity thus far, is that Canada should continue investing in scientific research through CERC and C150. Other countries are making large investments in research; therefore, in order to remain globally competitive, Canada needs to continue offering awards of similar calibre.

Evaluation Question 2: To what extent have the CERC and C150 programs attracted world-class researchers to Canada?

Through bibliometric analysis, this evaluation has determined that the CERC and C150 programs have indeed been successful in attracting world-class researchers to Canada. In turn, the reputation and innovative research of these chairholders has been cited as a main factor in attracting faculty and HQP to core teams, and has also facilitated the forming of partnerships and collaborations both nationally and internationally.

Evaluation Question 3: To what extent have the CERCs contributed to enhanced and sustainable research capacity at Canadian universities in areas of strategic importance identified by the federal government?

In the context of this evaluation, sustainability is operationalized as the growth and retention of the core team, partnerships and collaborations built through the CERC, and the continued prolific production of quality research outputs. An in-depth assessment of sustainability capturing longer-term impacts will likely only be possible in the context of the next CERC evaluation. However, it was still possible to gauge sustainability in a preliminary way in this evaluation by examining chairholders’ and core team members’ intent to remain at their host institution and/or in Canada, as well as the perceived impact of the end of CERC funding on partnerships and collaborations.

The majority of chairholders perceive the CERC program to have had great influence on their ability to establish both national and international partnerships and collaborations, which in turn have been useful in leveraging additional sources of funding and laboratory resources. Bibliometric analysis has found that CERC host institutions have seen significant increases in annual publications in the research area of the CERC as a direct result of the chairholders’ output (approximately an additional 10 articles per year or a 13.3% increase). The CERC host institutions are also well above their comparator Canadian and foreign institutions in annual publications. Although the measured increase in publications from pre- to post-award for individual chairholders is modest rather than pronounced (and commensurate with the relative increase observed among CRCs), these figures are likely an underestimate given that (1) CERC-funded research outputs (even from Competition 1 CERCs) are still emerging and (2) these data only capture the chairholder’s publications and not those of the CERC team as a whole. Finally, publication volume captured through bibliometric data is but one of many indices of research capacity and contributions.

Although it might be of interest to examine how the CERC program compares to other tri-agency programs with respect to cost per publication, such comparisons would likely result in misleading conclusions. Recall that a CERC is defined not only by the chairholder’s contributions, but more holistically by the core team and research program that is built at the host institution, as well as the larger research networks that the CERC team establishes. Qualitative data from the current evaluation suggest that the CERC program has resulted in increased research capacity at host institutions and has greatly influenced the career trajectories of team members, thus contributing to a range of successes that extend beyond the accomplishments of the chairholder alone. Although the broader, cascading impacts of the program could not be quantified, it is important to keep these larger contributions in mind when comparing the CERC program to other programs that may, by design, have a narrower reach.

Importantly, CERCs reported that their partnerships and collaborations are providing linkages primarily with other academic institutions rather than other organizations in the private and public sector. Additionally, CERCs reported a low prevalence of research outputs tailored to government and public policy contexts, primarily citing research outputs tailored to academic audiences (i.e., scholarly refereed journals and conference proceedings). The implication is that CERCs may not be reaching wider audiences beyond academia—an expected intermediate outcome as per the program’s logic model. Although this may in part be due to the longer time period required for government and public policy uptake, this evaluation indicates that increasing the visibility of the chairholders and their research would be beneficial, in turn introducing potential opportunities to establish linkages in other sectors and disseminate research to non-academic audiences, including government decision-makers.

In terms of sustaining research capacity in Canada, the majority of CERC chairholders (nearly 80%) plan to remain at their host institution following their CERC term; in addition, 50% of core team members surveyed indicated a desire to remain in Canada after the CERC term. CERC chairholders’ desire to stay at their host institutions is influenced by a number of factors, including the success and strength of the research program they have created, the investment in infrastructure they have made at the institution, the support and level of commitment to sustainability received from their host institution, and their ability to secure additional funding at the end of the CERC term. Host institutions report growth through the CERC program—namely evidenced by the CERC’s role in the development of new research programs, the creation of new faculty positions, the promotion of research more broadly and the development of new technologies. An additional key concern held by the majority of chairholders was in the overall sustainability of the CERC program and the potential impact that the end of CERC funding might have on their ability to sustain the collaborations and partnerships they have fostered through their position. The degree to which the program truly leads to the creation of sustainable research capacity should be more evident at the time of the next evaluation, at which time over five years will have elapsed from the end of the Competition 1 CERC terms.

Evaluation Question 4: To what extent are the design and delivery of the CERC and C150 programs effective and cost-efficient?

Several key strengths of the programs were noted by both chairholders and institutional representatives, including the flexibility in the use of CERC funds, the value and prestige of the CERC and C150 awards and the selection of the CERCs in strategic areas of research for Canada. However, interviews with chairholders and institutions alike revealed there was sometimes misalignment of expectations, largely a product of proposals and sustainability plans that lacked concrete goals and commitments.

Other concerns raised by chairholders and other key informants surrounded the length of the CERC term; that is, the number of years available to spend the $10 million award. Beyond the fact that the CERC is not renewable, several chairholders indicated that the seven-year term was too short a period to build such a large research program. It was also noted that research needs and timelines vary according to research area. Delays in getting research labs running at the start of the CERC terms was a challenge, especially when considering the high level of progress that is expected with this kind of program. Extending the terms or having a more explicitly defined tapering-off period would be helpful. Although automatic extensions of one year are provided and the Terms and Conditions of the award allow for the possibility of an additional extension with proper justification, the latter option was not universally understood among chairholders and institutions.

The timeframe for the CERC application and nomination process was considered too lengthy and onerous, which ultimately led to the loss of desirable candidates in favour of other job opportunities. In addition to the poor timing of the competition (i.e., over the summer while many people were away and difficult to reach), the primary issue with the C150 competition was the fact that its timeline was too compressed, which created a number of logistical issues and ultimately resulted in candidates declining the potential nomination due to the timeframe.

CERC chairholders from the first and second competitions are relatively homogeneous, generally not identifying with any of the four designated groups (i.e., women, persons with disabilities, Indigenous Peoples and members of visible minorities). However, several advances in the design of the CERC program (that were also applied to the C150 program) have been made since the first two CERC competitions to increase the level of equity and diversity within the program. Namely, the introduction of formal equity, diversity and inclusion (EDI) requirements in the selection criteria and institutional recruitment process, the inclusion of a detailed equity plan and the inclusion of an individual with EDI-related expertise on peer review panels. These changes to the recruitment and selection processes in the latest CERC and C150 competitions have resulted in a more diverse group of chairholders and core team members in terms of an increased representation of women and visible minorities among awardees.

Despite advances over the last few years, there are still a number of EDI implementation challenges, which in part pertain to the lack of clarity around EDI requirements and what recruitment targets should be applied across the various equity-seeking groups. Review panel members reported struggling with how to assess and weigh EDI considerations in the selection and review process. This was also a commonly perceived challenge of chairholders when recruiting core team members. In addition, review panel members expressed that the individual(s) invited to provide EDI-related advice was not used effectively within the peer review process. Finally, institutions lamented the overall lack of clarity regarding the required elements of an equity plan at the time of application.

Performance Reporting

As the CERC and C150 are relatively new programs, reporting practices have continued to evolve over time. Indeed, a tri-agency working group was formed in 2016 to further refine annual reporting templates. Although quantitative data extracted from annual reports were sufficient to support the evaluation of the program, the formulation and structure of key questions were often modified from year to year, which in many instances precluded longitudinal analysis. Additionally, based on wide variability in responses to certain items on the annual report combined with informal discussions with CERC chairholders and their administrative staff, there appeared to be a lack of universally applied definitions for key constructs (i.e., partnerships vs. collaborations; core team member; “providing expert advice”). Finally, there was a general perception by chairholders and institutional representatives that the annual reporting requirements were fairly onerous and lengthy, and that not all collected information was examined.

Appendix A – Program profiles

CERC program

The CERC program was launched in 2008 to support Canadian institutions with the goal to expand Canada's already growing reputation as a global leader in research and innovation. The program awards world-renowned researchers and their teams $10 million over seven years to establish ambitious research programs at Canadian institutions (Government of Canada, 2017d). Canadian institutions co-deliver the program with the three funding agencies, as each institution is required to provide matching funding and is responsible for each CERC unit.

Funding opportunity history

Since the inception of CERC in 2008, three competitions have been held, with launch start dates of 2008, 2012, and 2016. The first two competitions resulted in the award of 29 chairs across the country (the third competition is underway, with eight new chairholders announced in April 2019). Of the 29 chairs awarded in Competitions 1 and 2, three terminated their positions early (one from Competition 1, two from Competition 2), 18 from Competition 1 are now CERC laureates, and eight from Competition 2 are still active. Chair positions are selected in accordance with the priority areas determined by the Government of Canada. The first and second competitions were guided by the 2007 federal Science and Technology (S&T) Strategy, and focused on environmental science and technologies, natural resources and energy, health and related life sciences and technologies, information and communications technologies, and others (consisting of “open” priority areas).

Governance

A Steering Committee and a Management Committee oversee the CERC program. The Steering Committee is composed of the presidents of the three granting agencies (CIHR, NSERC and SSHRC); the deputy ministers of ISED and of Health Canada; and the President of the CFI, as an observer. The Steering Committee provides strategic direction for the program and makes final decisions on funding recommendations (Tri-agency Institutional Programs Secretariat, 2017a). The Management Committee is composed of the associate vice-president of TIPS; a representative at the vice-president level from each of the three granting agencies and from CFI (still as an observer); the director, Research Policy and Outreach Division at Health Canada; and the assistant deputy minister, Science and Research Sector at ISED (Government of Canada, 2017b). The Management Committee supervises the operation and coordination of the program administration, monitoring, and communications. It also approves the performance measurement strategy (PMS) (Government of Canada, n.d.). TIPS is in charge of the daily administration of the program and reports to the Steering Committee, the minister of Science, the Treasury Board of Canada Secretariat and the Parliament of Canada.

Application and review

CERC awards follow a two-phase selection process that involves multiple entities. Phase 1 consists of a preselection of eligible Canadian institutions based on their research proposal, while Phase 2 consists of a review of the chairholder nominations provided by the preselected institutions (Government of Canada, 2016e).

In order to be considered eligible for the Phase 1 application, Canadian institutions should have the authority to confer their own degrees and must receive a minimum of $100,000 a year in funding from at least one of the three federal granting agencies (Government of Canada, 2016e). The proposals are reviewed based on the following criteria:

• “the institution's research strengths in the proposed field, assessed against global standards of excellence;
• the promise of the proposed field of research for the Chair, measured in the context of leading global research in the proposed field, and the likelihood that the work associated with the proposed Chair will be recognized as globally relevant and will advance the frontiers of research in the field on a global scale;
• the extent to which the proposal fits in one or more of the priority areas identified or addresses other issues of benefit to Canada;
• the ability of the institution to sustain the research advantage created by the proposed Chair after the seven-year term of the Chair expires;
• the ability of the institution to leverage additional resources that, together with the CERC program, will enable the institution to adequately support the direct and indirect costs associated with a world-class program of research; and
• the potential to apply the research results from the Chair to advance public policy and/or the potential to commercialize research discoveries from the Chair” (Government of Canada, 2016e).

Phase 2 begins when the institutions shortlisted in Phase 1 are invited to nominate leading researchers as chairholders. Competitions are open to both Canadian and international nominees, but Canadian researchers may not be nominated by the institution in which they currently hold a position. Nominees can come from academia or other sectors based on the following:

• “Nominees must be full professors or associate professors who are expected to be promoted to the full professor level within one or two years of the nomination” (Government of Canada, 2016e).
• “If [the nominees] come from outside the academic sector, [they] must possess the necessary qualifications to be appointed at these levels” (Government of Canada, 2016e).

Additionally, the selection of chairholders is informed by the following criteria:

• the quality of nominees;
• nominees’ fit with the proposal submitted by the institution in Phase 1; and
• the quality of the institutional recruitment process, particularly the extent to which it ensures equity and diversity (new for Competition 3) (Government of Canada, 2016e).

As part of the Phase 2 review process of Competitions 2 and 3, institutions could submit, in tandem with their CERC program nomination, a proposal requesting infrastructure support from the CFI through the partnership stream of the John R. Evans Leaders Fund. The institution could request up to 40% of the total cost of the infrastructure project to a maximum of $800,000. In the context of Competition 1, institutions could submit requests for infrastructure support to the CFI, but through the unaffiliated stream of the John R. Evans Leaders Fund.

Chair operation

Chairs are administered through the institutions. CERC awards directly fund the costs of chairholders and their program of research. This includes the chairholders’ salary and the salaries of the CERC research team, excluding the salaries of faculty members who become part of the CERC team and are already remunerated by the university (Government of Canada, 2017j). Upon arrival at the host institution, chairholders form their core research team. This core team is expected to be composed of diverse faculty researchers and a range of HQP (e.g., graduate students, post-doctoral fellows, etc.). In addition, CERC research teams are expected to establish or further develop collaborations and partnerships. Chairholders and their teams are expected to contribute to important discoveries, insights and breakthroughs. This new knowledge is then expected to be broadly disseminated and accessible to achieve the greatest impact (Government of Canada, 2017j; Tri-agency Institutional Programs Secretariat, 2017a).

Reporting

Within the first year of being awarded the chair, the Secretariat conducts institutional visits. The main purpose of the visit is to meet with both the CERC chairholder and university representatives to ensure that the chairholder and the research program are being well established and are positioned to deliver on the expected outcomes (Tri-agency Institutional Programs Secretariat, 2017a).

Chairholders and institutions submit annual progress reports to TIPS by June 30 of each year. The chairholder report provides an account for various administrative elements and productivity indices. The institutions additionally provide an annual statement of account, as well as information on matched and leveraged support. As of 2016-17, host institutions have been required to report on their strategies and policies to promote EDI, and details of their sustainability plan for the program of research undertaken by the CERC (Tri-agency Institutional Programs Secretariat, 2017a, n.d.-d, n.d.-c).

In addition to annual reporting, a mid-term review of each CERC occurs in the fourth year of the CERC award. Continued funding is conditional on a successful mid-term review. The mid-term review focuses on two points: the progress of the CERC research program and the institutional component of the award.

• The progress of the CERC research program is assessed based on three main elements: the status of the research program (based on the research program proposal submitted in the application and the progress expected by mid-point); the quality and impact of research outputs of the CERC team (publications, conferences, patents, etc.); and the extent to which the chair has improved the institution’s research capacity and has fostered growth at the institution.
• The institutional component of the award is assessed by considering the matching funds/leveraged support secured by the institution and its plans for sustainability of the research advantage provided by the CERC funding.

Program expenditures

To date, there have been three CERC competitions. The program had an annual envelope of about $43.1 million before the third competition, with the annual envelope expected to increase by $2.5 million with the award of the new Competition 3 chairs. Operating expenditures are approximately $700,000 per year.

Table 1: CERC expenditures

Fiscal year	2016-17	2017-18	2018-19	2019-20	2020-21
Budget 2008 (ongoing)	$28,600,000	$28,600,000	$28,600,000	$28,600,000	$28,600,000
Budget 2011 (ongoing)	$14,500,000	$14,500,000	$14,500,000	$14,500,000	$14,500,000
Budget 2016 ($20M over 8 years)	-	-	$2,500,000	$2,500,000	$2,500,000
Total	$43,100,000	$43,100,000	$45,600,000	$45,600,000	$45,600,000

Source: TIPS, 2017

---

EDI

All 19 chairholders nominated in the inaugural competition were men. This lack of diversity among CERC recipients prompted the minister of Industry to mandate a panel on CERC Gender Issues to examine the gender disparity and render recommendations (Ad Hoc Panel on CERC Gender Issues, 2010). As a result, CERC management introduced subsequent changes to the program before the second competition. Four measures were implemented to allow a more diverse pool of chairholders:

• Universities were asked to report on their recruitment and outreach efforts in both Phase 1 and Phase 2 of the competition, and the quality of recruitment was formally added as a selection criterion at Phase 2.
• Expert reviewers were informed that both rising stars and established researchers were eligible.
• It was agreed that shortlisted proposals would match the number of available chairs, so there would be less uncertainty for nominees.^{Footnote 47}
• Four out of 11 chairs announced for the second competition were open to all areas of research (Science-Metrix, 2014).

Additionally, new equity and diversity requirements were included in the third competition. Universities are now required to include a detailed equity plan and recruitment strategy in the Phase 1 application that promotes the participation of women and other underrepresented groups within the CERC team as part of the application process (Tri-agency Institutional Programs Secretariat, 2016, 2017a).The CERC team members are also asked to complete self-identification forms that will allow TIPS to determine whether the EDI practices of institution are having an impact (Tri-agency Institutional Programs Secretariat, n.d.-e).

Logic model

CERC logic model long description

This logic model is a visual illustration of the program’s inputs, activities, outputs and the expected immediate, intermediate, long-term and strategic outcomes. The model is split into the following sequenced sections, starting from the bottom left to the top left.

1. Inputs, which include:
   1. Government of Canada funding for CERC program that builds on past investments in postsecondary research; and
   2. Government of Canada policies and priorities in Science, Technology, and Innovation.
2. Activities (operations using inputs, to produce outputs), which include:
   1. design and, based on lessons learned, modify process to establish Canada Excellence Research Chairs at Canadian universities in areas of strategic importance for Canada;
   2. provide advice to institutions, nominees, and chairholders before, during, and after the competition;
   3. manage rigorous selection process to determine which institutions may establish Canada Excellence Research Chairs (approval of nominations); and
   4. monitor active chairs and communicate program outcomes.
3. Outputs, which include:
   1. program literature, communication products, and guidance to institutions and chairholders; and
   2. grants to Canadian universities.
4. Immediate outcomes (expected to begin immediately, i.e., in zero to two years, but may continue throughout the duration of the award), which include:
   1. world-class researchers are attracted to Canada to take up CERC;
   2. universities invest in areas of strategic importance; and
   3. high-calibre and diverse teams are established in CERC areas.
5. Intermediate outcomes (expected to result from the implementation of the immediate outcomes and may extend for the duration of the award, i.e., one to seven years after the start of the award), which include:
   1. universities grow in areas of strategic importance to Canada;
   2. ambitious programs of research lead to important discoveries and insights;
   3. new collaborations and partnerships are developed and existing ones are strengthened;
   4. the chairholders and the CERC core team share their expertise with knowledge users; and
   5. students, postdoctoral fellows, and other HQP increase in numbers, gain expertise and are offered adequate support.
6. Long-term outcomes (expected to occur after the end of the award, i.e., 7 to 10 years after the start of the award), which include:
   1. institutions who held CERCs maintain their research advantage;
   2. Canada is recognized internationally as a global destination of choice for research;
   3. individuals and organizations from outside of the academic sector benefit from the research conducted by CERCs; and
   4. Canada increases its research capacity in areas of strategic importance.
7. Strategic outcomes (the program contributes to the three granting agencies achieving their strategic outcomes), which include:
   1. Canada is a world leader in the creation, dissemination and application of health research knowledge; advancing, connecting and applying new knowledge in the natural sciences and engineering; and social sciences and humanities research and research training.

Assumptions

• Past and current investments made by governments, universities and other sectors have made Canada attractive to world-class researchers.
• Through strategic research plans, universities have established priorities and become internationally competitive in areas of strategic relevance to Canada.
• World-class researchers and their families are willing to move to Canada.
• Universities engage and participate as intended in the process from application to continued investment in established Canada Excellence Research Chairs throughout the award.
• Sufficient funding in Canada through a variety of sources is available to be secured to maintain the research advantage after the seven years of funding are over.

---

C150 program

A $117.6 million investment for the Canada 150 (C150) Research Chairs program was announced in April 2017 in celebration of Canada’s 150th anniversary (Innovation, Science and Economic Development Canada, 2017). The objective of the C150 program was to build on the gains made through other tri-agency programs including CERC, CRC and CFREF, further strengthening Canada’s research capacity by attracting top-tier internationally-based scholars and researchers (including Canadian expatriates) (Government of Canada, 2017e; Innovation, Science and Economic Development Canada, 2017). To this end, two potential award values were offered: $350,000 or $1 million annually, both for a term of seven years (Government of Canada, 2017e). Similar to CERC, C150 supports Canadian universities in their efforts to build on Canada's growing reputation as a global centre for innovation, science and research excellence. However, in contrast to CERC, C150 was announced as a one-time investment, with the selection of chairs limited neither by priority area nor by career stage (Government of Canada, 2017i). The specific objectives of the program are to (Government of Canada, 2017j; Tri-agency Institutional Programs Secretariat, 2017a) (Government of Canada, 2017g; Tri-agency Institutional Programs Secretariat, 2017):

• strengthen Canada’s ability to attract top-tier international scholars and researchers; and
• enhance Canada’s reputation as a global centre for science, research and innovation excellence.

Funding opportunity history

The C150 Research Chairs Competition was officially launched on June 21, 2017 and resulted in the appointment of 24 chairholders (Government of Canada, 2017g). Subsequently, one chairholder resigned from the position and was replaced by a new chairholder from the reversion list in July 2018, marking the conclusion of the C150 Research Chairs competition.

Application and review

In contrast to the two-phase CERC application and review process, the C150 competition was completed in a single application and review phase. However, about a month before the deadline for the submission of complete applications, institutions were required to submit a registration form providing key information about each application they intended to submit (Government of Canada, 2017i). Mandatory components of the full applications were as follows:

• Application form and attachments
  • Full CV
  • Budget form
  • Demonstration of how the application meets the program’s selection criteria
  • Equity, diversity and inclusion attachment
• Application administrative form
• Nominee self-identification form
• Three letters of reference to demonstrate how the individual meets the evaluation criteria for the program.

The letters of reference had to fulfill the following requirements:

• All three were expected to be established authorities in the field who were not in a conflict of interest with the candidate as per the Conflict of Interest and Confidentiality Policy of the Federal Research Funding Organizations. To this end, the referee had to provide a description of their professional relationship with the nominee and declare that there was no conflict of interest.
• At least one had to be provided by a recognized international authority in the candidate’s field who does not reside in the country where the candidate was working at the time of application.
• All three were expected to limit unconscious bias and address how the applicant meets the program evaluation criteria.

The C150 competition received applications from eligible Canadian institutions only. In order to be considered an eligible Canadian institution, an organization had to be recognized as a degree granting institution and have received at least an annual average of $1 million in funding from at least one of the three federal granting agencies (Government of Canada, 2017i). In order to be eligible for nomination, researchers had to be working and residing outside of Canada (nationality had no impact on their eligibility to be nominated). No individual is permitted to hold a Canada 150 Research Chair and either a Canada Research Chair or a Canada Excellence Research Chair at the same time (Government of Canada, 2017i).

All applications were peer reviewed by experts in the candidate’s field of research. Subsequently, a multidisciplinary expert review panel composed of world-leading national and international researchers evaluated all applications based on the selection criteria, taking into consideration the expert reviewers’ written assessments and the application materials. Based on overall rankings, the expert review panel made recommendations, for ratification and approval, to the Tri-agency Steering Committee regarding the chairs that should be funded. A ranked reversion list was also produced in the event that any awards were declined (Government of Canada, 2017i). Ranking of applications was established through holistic considerations of the selection criteria as opposed to a fixed weight approach. The four selection criteria were:

• Research/academic merit of the nominee
• Quality of the institutional support
• Institutional equity and diversity considerations
• Potential contribution of the proposed chair to the excellence of the Canadian research ecosystem.

As part of the C150 application process, institutions could submit a proposal for infrastructure support from the CFI through the partnership stream of the John R. Evans Leaders Fund. The institution could request up to 40% of the total cost of the infrastructure project to a maximum of $800,000.

Chair operation

Chairs are administered through the institutions. C150 awards directly fund the costs of chairholders and their program of research. This includes the chairholders’ salary, as well as the salaries of the C150 research team, excluding the salaries of other faculty members (Government of Canada, 2017j). Up to 25% of the amount allocated for direct costs of research may be used to cover indirect costs in conformity with eligible expenses from the Research Support Fund (Tri-agency Institutional Programs Secretariat, 2017b).

The funding provided through the program is expected to allow universities to attract world-class researchers who would not otherwise have come to Canada (Tri-agency Institutional Programs Secretariat, 2017b). In similar fashion to the CERC program, chairholders are expected to begin forming a core research team upon arrival. This core team is expected to be composed of diverse faculty and HQP (i.e., graduate students, post-doctoral fellows, research technicians, etc.), recruited and selected based on EDI best practices. Chairholders and their teams are expected to engage in programs of research recognized internationally by peers as having the potential to achieve high impact. The work of the team is expected to lead to impactful and innovative discoveries and insights that contribute to their discipline (Government of Canada, 2017j; Tri-agency Institutional Programs Secretariat, 2017b). In addition, C150 Research Chairs are expected to establish or further develop cross-sectoral collaborations for themselves and for the host institution. Contrary to the CERC program, matching funds from external partners is not a requirement. However, it is expected that institutions will offer additional financial support to complement the C150 funding (Government of Canada, 2017i; Tri-agency Institutional Programs Secretariat, 2017b).

Reporting

Performance information is collected at several points to make selection decisions, to monitor the performance of the individual chairholders, to monitor the financial practices of the institutions, and to monitor the performance of the C150 Research Chairs program itself. Several of the data sources are found in the C150 Research Chairs application tracking database, containing information drawn from nomination packages. Statement of accounts must be submitted by all recipient institutions by June 30 of each year, reflecting activities for the previous fiscal year (Tri-agency Institutional Programs Secretariat, 2017b).

Institutions and chairholders are also required to submit annual reports. The chairholder annual report is used to collect information on the chairholder’s activities, research team, progress towards research goals, research contributions, and new awards or accolades they have received in the previous year.  The institutional annual report collects information on institutional growth in the area of research of the C150 Research Chair, diversity within the core research team, commitments to ensure post-chair funds for the long-term sustainability of the chair and commitment to EDI (Tri-agency Institutional Programs Secretariat, n.d.-a).

Program expenditures

The C150 Research Chair program was announced as a one-time competition with a $117.6 million budget spread over eight fiscal years. The program had a $4.0 million budget in 2017-18, which is expected to rise to $16.8 million in 2018-19 once all of the Chairs have been awarded (Tri-agency Institutional Programs Secretariat, 2017b). Just over $7 million or 5.96% is planned to be dedicated to operating expenditures over the course of the program.

Table 4: Projected C150 program expenditures

Fiscal year	2017-18	2018-19	2019-20	2020-21	2021-22	2022-23	2023-24	2023-24
Total – Fiscal Framework	$4,000,000	$16,800,000	$16,800,000	$16,800,000	$16,800,000	$16,800,000	$16,800,000	$12,800,000

Source: (Tri-agency Institutional Programs Secretariat, 2017b)

---

EDI

All institutions that submitted nominations were also required to demonstrate that their recruitment and selection processes were based on best practices consistent with the government's larger EDI strategy. As such, all applications required a discussion of how the institution had fulfilled the following elements:

• An open advertisement that includes a strong and meaningful commitment statement to equity and diversity, encouragement for persons in the four designated groups to apply, acknowledgement of the potential impact that career interruptions can have on a candidate’s record of research achievement and encouragement for applicants to explain how career interruptions might have impacted their record of research achievement.
• A candidate search process that puts significant efforts into identifying a diverse pool of potential applicants and an evaluation process that ensures that applicants with career interruptions are not unfairly disadvantaged.
• A search committee with diverse members who have been trained on the potential negative impact that unconscious bias can have on the career path of individuals from the four designated groups.

Logic model

C150 logic model long description

This logic model is a visual illustration of the program’s inputs, activities, outputs and the expected immediate, intermediate, long-term and strategic outcomes. The model is split into the following sequenced sections, starting from the bottom left to the top left.

1. Inputs (financial and non-financial resources used to deliver activities, produce outputs and accomplish outcomes), which include:
   1. Government of Canada funding for C150 Research Chairs program
2. Activities (operations using inputs to produce outputs), which include:
   1. design processes to manage the C150 Research Chairs program;
   2. manage rigorous selection process to determine which institutions may establish C150 Research Chairs (approval of nominations); and
   3. monitor active chairs and communicate program outcomes.
3. Outputs (direct products/services that stem from activities), which include:
   1. program literature, communication products, and guidance to institutions and chairholders; and
   2. grants to Canadian universities.
4. Immediate outcomes (expected to begin immediately, yet may continue through the duration of the award, i.e., one to two years after the start of the award), which include:
   1. top-tier international researchers are attracted to take up C150 Research Chairs; and
   2. high-calibre, diverse research teams are established.
5. Intermediate outcomes (expected to result from the implementation of the immediate outcomes and may extend for the duration of the award, i.e., two to seven years after the start of the award), which include:
   1. discoveries and insights are produced;
   2. collaborations are created/strengthened; and
   3. training opportunities and institutional support for HQP are enhanced.
6. Long-term outcomes (expected to occur after the end of the award, i.e., eight to 12 years after the start of the award), which include:
   1. Canada is recognized as a global centre for science, research and innovation excellence; and
   2. Canada increases its research capacity in areas of C150 Research Chairs.
7. Strategic outcomes (the program contributes to the three granting agencies achieving their strategic outcomes), which include:
   1. Canada is a world leader in the creation, dissemination and application of health research knowledge; advancing, connecting and applying new knowledge in the natural sciences and engineering; and social sciences and humanities research and research training.

---

Appendix B – Evaluation matrix

Evaluation Question	Purpose of the Evaluation Question	Indicators	Data Sources
Relevance
1. To what extent do the CERC and C150 programs address (or continue to address) a unique need?	Building on the findings from the last evaluation of the CERC program, the evaluation will include a brief assessment of the continued need for the programs within the suite of federal programs aimed at building research capacity by attracting or supporting the attraction of world-class researchers. This need will also be assessed vis-à-vis C150, similarly designed to attract strong research capacity to Canada. This assessment involves examining key program features and stakeholder perceptions. The analysis will also identify key changes to the national and international context that have impacted the relevance of programs occupying this niche. The evaluation did not intend to update the comparison study of other international programs that was conducted in the last CERC evaluation. Accordingly, there are no new findings related to how competitive the programs are on a global scale based on their value and duration.	a. Brief description of the niche of CERC and C150 in relation to other federal programs aimed at attracting or supporting the attraction of world-class researchers (e.g., CRCs, CFREF) with reference to key program features such as the programs’ detailed objectives, funding amounts, and targeted/non-targeted funding	Document and literature review
		b. Perceptions of the niche of the CERC and C150 programs in relation to other federal programs aimed at building research capacity	Interviews (VPs of research, active CERC chairholders from the second competition, C150 chairholders, selection committee and review panel members) Case studies (chairholders, representatives from the university’s research office and/or the dean of the faculty in which the CERC is housed)
		c. The extent to which recent changes have occurred in the national and international context that have impacted the relevance of programs occupying the identified niche (since the last CERC evaluation in 2013-14)	Document and literature review Interviews (VPs of research, C150 chairholders, selection committee and review panel members) Case studies (representatives from the university’s research office and/or the dean of the faculty in which the CERC is housed)
Performance
2. To what extent have the CERC and C150 programs attracted world-class researchers to Canada?	This evaluation question focuses on the productivity and scientific impact of the CERC and C150 chairholders before their nomination to assess if world-class researchers were in fact attracted as per the goal of the programs. The diversity of chairholders from both programs was also examined, with a specific focus on EDI groups. Finally, C150 chairholders were interviewed and asked to what extent the C150 program has made a difference in attracting world-class researchers that would otherwise not have come to Canada. Similar information was solicited from CERC chairholders during the previous evaluation.	a. Scientific production and quality: Median, HCP10%, and HCP1% of CERC and C150 chairholders in the 10-years before nomination (relative to matched Canadian and foreign CRCs)	Bibliometric study
		b. Extent of diversity of CERC and Canada 150 chairholders	CERC and C150 administrative data review
		c. Perception of the extent to which the CERC and C150 programs have attracted world-class researchers that would otherwise not have come to Canada	Interviews (C150 chairholders, VPs of research, selection committee and review panel members) Document and literature review (previous evaluation of CERC)
		d. Extent to which recent changes have occurred in the international context that could have an impact on the programs’ ability to attract world-class researchers (since 2013-14)	Interviews (VPs of research, C150 chairholders, selection committee and review panel members) Case studies of CERCs (core team members [faculty])
3. To what extent have the CERCs contributed to enhanced and sustainable research capacity at Canadian universities in areas of strategic importance identified by the federal government?	The assessment of CERC will focus on its contribution to enhanced and sustainable research capacity in strategic research areas identified by the federal government. The term “research capacity” can have a broad scope; related indicators attempt to only capture the key aspects of research capacity of interest to stakeholders consulted during the planning-phase of the evaluation. Namely, the featured indicators include the chairholder’s success in building research capacity through the core team, the impact of the CERC program on the chairholder’s ability to establishing new collaborations and partnerships across sectors, CERC team research outputs and knowledge mobilisation activities, and institutional growth as a result of the CERC.	High-calibre and diverse core teams
		a. Number, type, and role of CERC core team members	CERC administrative data and file review (annual institution and chairholder reports, mid-term review) Survey of CERC core team members Lists of core team members provided by CERCs
		b. Scientific production and quality: Median ARC (Average Relative Citation), HCP10% (Highly Cited Publications), and HCP1% of CERC chairholders in the post-award period (compared with matched foreign and Canadian CRCs)	Bibliometric study
		c. Number and type of research outputs of CERC core teams (e.g., publications, patents, awards, invited engagements)	CERC administrative data and file review (annual chairholder reports, mid-term review) Survey of CERC core team members
		d. Extent of other funding obtained from chairholders	CERC administrative data and file review (annual institution and chairholder reports, mid-term review)
		e. The extent to which host institutions, chairholders, and CERC core teams have implemented adequate measures to mitigate systemic barriers in their recruitment and nomination processes (i.e., they have identified key systematic barriers and have EDI plans in place)	Document and literature review Interviews (VPs of research, active CERC chairholders from the second competition, CERC chairholders who left before the end of their term, selection committee and review panel members) Case studies of CERCs (chairholders, current and former core team members [faculty, HQP], representatives from the university’s research office and/or the dean of faculty in which the CERC is housed) Survey of CERC core team members
		f. Proportion of current chairholders and CERC core team members self-identifying in the four designated groups	CERC administrative data and file review (self-identification forms) Survey of CERC core team HQP
		g. Extent and nature of HQP core team training experiences (e.g., type of involvement and training, opportunities to develop new expertise and enhance existing skillsets)	Case studies of CERCs (particularly interviews with current and former core team members) Survey of CERC core team HQP
		h. Number and proportion of HQP employed in an area related to the CERC following their involvement with a CERC core team, in Canada or abroad	Survey of CERC core team HQP
		i. Percentage of HQP who are in Canada and intend to remain, or intend to return	Survey of CERC core team HQP
	This subsection will focus on the number and nature of new and existing collaborations with academics and other sectors. This information will be derived from existing performance data and supplemented by case studies or interviews with chairholders.	Collaborations and partnerships
		a. Number of new and existing collaborations and partnerships initiated during the grant term (national/international, sector, etc.)	CERC administrative data and file review (annual chairholder reports, mid-term review)
		b. Descriptions of the nature of new and existing collaborations and partnerships (e.g., cash/in-kind contributions, type and level of involvement)	CERC administrative data and file review (annual chairholder reports, mid-term review) Interviews (active chairholders from the second competition) Case studies of CERCs (Chairholders)
		c. Amount of cash and in-kind investment from partners	CERC administrative data and file review (annual report for chairholders, mid-term review)
	This subsection will feature examples of the scientific impact of discoveries and insights from the CERC core teams and the provision of expert advice by chairholders. Program-collected performance data, key informant interviews, and case studies will address the indicators in this subsection.	Knowledge mobilization
		a. Examples of discoveries and insights that have had a wide scientific impact	Interviews (active CERC chairholders from the second competition) Case studies (chairholders, core team members, representatives from the university’s research office and/or the dean of faculty in which the CERC is housed) Document and literature review
		b. Examples of chairholders who have provided expert advice to potential knowledge users	Interviews (active chairholders from the second competition) Case studies of CERCs (chairholders) CERC administrative data and file review (annual institutional and chairholder reports)
	This subsection will examine the growth of the CERCs and their sustainability. Program-collected performance data, key informant interviews, and case studies will provide key evidence.	Investment and growth in research areas identified by the federal government
		a. Number of chairholders who are also involved in one or more CFREFs and/or are a recipient of other federal funding (e.g., CFI or Genome Canada) or other non-federal funding	Document and literature review
		b. Number and proportion of institutions with a CERC that also have a CFREF in the same area of research	Document and literature review
		c. Examples of institutional growth in the strategic research areas identified by the federal government (e.g., new faculty positions, new programs in areas related to the research of the chairholder, new infrastructure, etc.)	Interviews (CERC chairholders, selection committee and review panel members) Document and literature review CERC administrative data and file review Case studies (chairholders, representatives from the university’s research office and/or the dean of faculty in which the CERC is housed)
		d. The extent to which institutions can be anticipated to retain a critical mass of outstanding tenured and non-tenured key researchers, and HQP, in the same research area	Interviews (VPs of research, active CERC chairholders from the second competition) Case studies (chairholders, core team members [current and former HQP, faculty], representatives from the university’s research office and/or the dean of faculty in which the CERC is housed) CERC administrative data and file review (sustainability plans, mid-term review)
Design, Delivery, and Efficiency
4. To what extent are the design and delivery of the CERC and C150 programs effective and cost-efficient?	Certain aspects of the programs’ design and delivery are examined to highlight areas of strength and areas for improvement. This will include comparing the design of the competition cycle for the two programs, term length of CERC, and other facets of how the CERC is being administered. The evaluation also examines the extent to which the programs support EDI policy. A cost-efficiency analysis will examine the ratio of administrative expenditures in relation to the total amount of grant expenditures for CERC compared with that of other programs administered by TIPS.	a. Perceptions regarding the extent to which the programs are designed and delivered in an efficient and effective manner (including suggestions for improvement)	Interviews (VPs of research, active CERC chairholders from the second competition, former CERC chairholders who left before the end of their term, C150 chairholders, C150 nominees who declined the grant, selection committee and review panel members) Case studies (chairholders, representatives from the university’s research office and/or the dean of faculty in which the CERC is housed)
		b. Perceptions of the relative effectiveness of the competition cycles for CERC vs. C150	Interviews (VPs of research, active CERC chairholders from the second competition, C150 chairholders, C150 nominees who declined the grant, selection committee and review panel members) Case studies (chairholders, representatives from the university’s research office and/or the dean of faculty in which the CERC is housed)
		c. The extent to which CERC and C150 programs’ design and delivery support EDI (including description of recent modifications to CERC)	Interviews (C150 chairholders, C150 nominees who declined the grant, selection committee and review panel members) Document and literature review
		d. For CERC, the operating ratio of administrative costs to grant funding	Program and secretariat financial data

---

Appendix C – Evaluation methodologies

The evaluation of the CERC and C150 programs was based on multiple lines of evidence that included secondary data maintained by SSHRC, as well as primary data collection with stakeholders using qualitative and quantitative methods. Each line of evidence is described below.

Purpose	Scope/Sample
Document and literature review
Assisted in developing a strong understanding of the CERC and the C150 programs including their mandates, objectives, design and delivery. Also addressed aspects of the evaluation questions of relevance, performance and cost-efficiency.	A number of pertinent documents were reviewed, including performance measurement strategies, performance information profiles, and chairholder and institutional progress reports. The literature review portion focused on the most pertinent publications available to characterize and describe recent changes in the Canadian and international research context that have impacted the programs’ ability to attract world-class researchers and build research capacity in areas of strategic importance for the federal government.
CERC and C150 administrative data and file review
The purpose of the CERC and C150 administrative data and file review was to highlight descriptive statistics pertaining to the demographic composition of CERC and C150 nominees and active chairholders, to address the evaluation questions around cost-efficiency and to provide indices of productivity and capacity building in the case of the CERC program (e.g., research outputs, knowledge mobilization, collaborations, etc.).	For both C150 and CERC programs, demographic data (including EDI measures) on award nominees and recipients were available. Data were also available from CERC annual reports (both institutional and chairholder versions) for the evaluation period.
Case studies of CERCs (N = 9)
The case studies, featuring a sample of CERCs from Competition 1 who had completed or were nearing completion of their term, examined the relevance, performance, design and delivery of the CERC program, while also assessing the degree to which CERCs and their teams had built sustainable capacity at the host institution. Unintended impacts of the program were also examined.	Nine case studies were completed, representing a sample of the 18 CERCs from the first competition. The case studies included a review of documents (i.e., CERC application files, sample publications, interviews, etc.), a review of the case studies completed in the last evaluation of the CERC program, and up to 10 interviews with the chairholder, representatives from the university’s research office, the dean of faculty associated with the CERC, faculty from the CERC core team, and current and former HQP from the CERC core team.
Key informant interviews (N = 51)
Stakeholders were selected to obtain their perspectives on a wide range of evaluation questions and indicators and to provide any insight on unintended impacts of the programs. These stakeholders help to corroborate, explain and further elaborate on findings from other data sources and are key to understanding why outcomes have or have not been achieved.	Interviews were conducted with CERC chairholders from the first and second cohorts, former CERC chairholders (i.e., those who left before the end of their terms); vice presidents of research; selection committee and review panel members; C150 chairholders; and C150 nominees who declined the award.
Survey of CERC core team members
Surveys provide efficiency in obtaining information from large groups of stakeholders, allow for the quantification and aggregation of a large volume of data and afford the ability to obtain outcome information directly from relevant stakeholders. This survey focused on faculty and HQP (current and former) from the active CERC core teams from Competitions 1 and 2. In addition to gathering information on the demographic composition of the CERC core teams, the survey focused on collecting quantitative data on the degree to which the CERC offered unique, career-enhancing opportunities to its core members and built capacity at its host institution.	A mixed-mode online survey with telephone follow-up was used to gather information from core team members, past and present, associated with CERCs from Competitions 1 and 2. A total of 562 CERC core team members completed the survey, resulting in an overall response rate of 37.1%. Membership from each CERC team was represented on the survey, with a similar proportion of faculty (41.6%) and HQP (38.3%) from the available sample frame electing to complete the survey.
Bibliometric analysis
Bibliometric analysis contributed to this evaluation by demonstrating the effectiveness of the CERC and C150 programs in attracting world-class researchers to Canada. The analysis also provided insight into the productivity of chairholders and the extent to which the CERC grants contributed to enhanced research capacity at Canadian universities in areas of strategic importance identified by the federal government.	First, the CERC program was examined with regard to its ability to attract world-class researchers to Canada. This was accomplished through two comparative analyses. The first analysis compared the prior performance of CERCs and unsuccessful applicants. The second analysis compared the prior performance of successful CERC applicants with a matched group of Tier 1 CRC holders (CRC1s). Second, the extent of contributions made by the CERCs to enhance and sustain the research ability of Canadian institutions was examined by comparing CERC and CRC1 holders in the post-award period. The last part of this analysis focused on the effect the CERCs have on their host institutions by comparing the profile of those institutions with that of similar international and Canadian institutions in the post-award period. Third, the ability of the C150 program to attract world-class researchers to Canada was examined by comparing the prior performance of C150 chairholders versus unsuccessful C150 applicants and matched CRC holders.

---

Appendix D – References

Advisory Panel on Federal Support for Fundamental Science. (2017). Investing In Canada’s Future: Strengthening the Foundations of Canadian Research.

Arim, R., & Frenette, M. (2019). Are mental health and neurodevelopmental conditions barriers to postsecondary access? Ottawa, ON: Statistics Canada.

Canada Excellence Research Chairs. (2015, July 3). Largest-ever Map of the Human Interactome Predicts New Cancer Genes. Retrieved April 18, 2018.

Canadian Association of University Teachers (CAUT; 2018, April). Underrepresented and underpaid: Diversity and equity among Canada’s post-secondary education teachers. Ottawa, ON: Author.

Canadian Mental Health Association. (2019). Stigma and discrimination. Toronto, ON: Author.

Council of Canadian Academies. (2018). Competing in a Global Innovation Economy: The Current State of R&D in Canada. Retrieved from Expert Panel on the State of Science and Technology and Industrial Research and Development in Canada website.

Dowdeswell, E., Fortier, S., & Samarasekera, I. (2010). Report to the Minister of Industry of the Ad Hoc Panel on CERC Gender Issues. Ottawa (ON): Industry Canada.

Gallop, C. J. (2016). Supporting success: Aboriginal students in higher education. Canadian Journal of Higher Education, 46(2), 206-224.

Goss Gilroy Inc. (2016). Evaluation of the Canada Research Chairs Program: Final Report.

Government of Canada. (2012). How global demographic and economic trends might affect Canada’s immigration program.

Government of Canada. (2016a, October 19). Canada Excellence Research Chairs Play Significant Role in Canada First Research Excellence Fund initiatives. Retrieved February 5, 2018.

Government of Canada. (2016b, November 25). Equity and Diversity Practices. Retrieved February 5, 2018.

Government of Canada. (2017, July 31). Application and Nomination Process. Retrieved from Canada 150 Research Chairs website.

Government of Canada. (2018, May 3). Program Statistics.

Hewlett, S. A., Marshall, M., & Sherbin, L. (2013, December). How diversity can drive innovation. Harvard Business Review.

Krisch, J. A. (2014, November 26). Protein Project Could Help Expose Cancer’s Causes. Scientific American.

Mohamed, T., & Beagan, B. L. (2019). "Strange faces" in the academy: Experiences of racialized and Indigenous faculty in Canadian universities, 22, 338-354. doi: 10.1080/13613324.2018.1511532

O’Doherty, H. (2018, April). Canada reduces barriers to disabled immigrants.

Reinhart, R., et al. (2019). The 10,000 PhDs at the University of Toronto: Using employment outcome data to inform graduate education. Plos One, 14.

Staniland, N. A., & Harris, C., & Pringle, J. K. (2019). "Fit" for whom? Career strategies of Indigenous (Maori) academics. Higher Education. doi: 10.1007/s10734-019-00425-0

Science-Metrix. (2010, December 8). Tenth-Year Evaluation of the Canada Research Chairs Program.

Science-Metrix. (2014). Evaluation of the Canada Excellence Research Chairs (CERC) Program: Final Evaluation Report.

Social Sciences and Humanities Research Council. (2019). Best practices in equity, diversity and inclusion in research. Ottawa, ON: Government of Canada.

Tri-agency Institutional Programs Secretariat. (2014, May 8). Performance Measurement Strategy for the Canada Research Chairs (CRC) Program.

Tri-agency Institutional Programs Secretariat. (2017a). Performance Measurement Strategy for the Canada Excellence Research Chairs (CERC) Program.

Tri-agency Institutional Programs Secretariat. (2017b, March). Performance Measurement Strategy for the Canada First Research Excellence Fund (CFREF).

Tri-agency Institutional Programs Secretariat. (2017c, May). Performance Measurement Strategy for the Canada 150 Research Chairs (C150 Research Chairs) Program.

Tri-agency Institutional Programs Secretariat. (n.d.). Canada Excellence Research Chairs (CERC)—Chairholder Annual Report.

Wang, J., Veugelers, R., & Stephan, P. E. (2015). Bias Against Novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators. Retrieved July 12, 2019.

Wright, T. (2018, April). After 40 years, federal government ending barriers to disabled immigrants.