---

List of Tables

Table 1:	Evaluation issues and questions
Table 2:	Examples of research excellence
Table 3:	Examples of dissemination and use of research results
Table 4:	Percentage of RDI/IDG respondents indicating new research questions, tools, methods, or theoretical approaches were developed for their project, disaggregated by funding status
Table 5:	In the words of trainees
Table 6:	Average cost-efficiency ratios for IG, IDG and comparable programs, 2005-06 to 2014-15
Table 7:	RDI/IDG applications, awards and grant expenditures, 2009-10 to 2014-15
Table 8:	SRG/IG applications, awards and grant expenditures, 2009-10 to 2014-15
Table 9:	Comparison of SRG and IG funding opportunities
Table 10:	Comparison of RDI and IDG Funding Opportunities

List of Figures

Figure 1:	Evaluation Scope
Figure 2:	Average of Relative Impact Factors of Countries with the Most SSH Publications, 2005-2013
Figure 3:	Average Percentage of Administrative Expenditures to Total Costs for IG, IDG and Comparable Programs, 2005-09 to 2014-15
Figure 4:	Principal Investigator Scholar Type, by Application Status for SRG, IG and IDG, 2009-2014

List of Acronyms

EAC	Evaluation Advisory Committee
CSP	Corporate Strategy and Performance
DG	Discovery Grants (NSERC funding opportunity)
HQP	Highly qualified personnel
IG	Insight Grants
IDG	Insight Development Grants
NSE	Natural sciences and engineering
NSERC	Natural Sciences and Engineering Research Council of Canada
PA	Program Architecture
RDI	Research Development Initiatives
SSH	Social Sciences and Humanities
SSHRC	Social Sciences and Humanities Research Council of Canada
SRG	Standard Research Grant

---

Executive Summary

Overview

This report presents the results of an evaluation of the Social Sciences and Humanities Research Council’s (SSHRC) Insight Grants (IG) and Insight Development Grants (IDG). The evaluation was conducted to meet the coverage requirements of the Treasury Board’s Policy on Evaluation (2009) and the requirements of the Financial Administration Act.

The evaluation used multiple lines of evidence to address the five core evaluation issues stipulated in the Policy on Evaluation (2009), which fall within two broad categories: relevance and performance. Relevance issues were assessed from 2010-11 to 2014-15, while performance issues, especially the achievement of expected outcomes, were assessed as of 2005, including for grants funded through the IG/IDG’s predecessors: Standard Research Grants (SRG) and Research Development Initiatives (RDI). Nine evaluation methods were used to collect, analyse and synthesize data across multiple lines of evidence: a document review, a literature review, an administrative data review (including applications and final research reports), a cost-efficiency analysis, key informant interviews, a researcher survey, a student survey, survey focus groups, and case studies.

Insight Grants provide funds to scholars working as individuals or in teams for long-term social sciences and humanities (SSH) research initiatives. For the 2012-2014 competition years, these grants provided between $7,000 and $500,000 over three to five years. Insight Development Grants support the development of new research questions, and experimentation with new methods, theoretical approaches or ideas in the SSH. These grants are valued at between $7,000 and $75,000 over one to two years and are allocated across two categories of researchers: established and emerging scholars.

Conclusions

Relevance

There is an ongoing need for SSH research to answer questions on issues that matter most to Canadians, such as those relating to culture, business, education, health, the environment and policy. SSHRC’s IG and IDG are two of the main funding opportunities currently available to SSH researchers in Canada, and the few alternatives that were identified had more limited accessibility and scope. Without IG/IDG funding, proposed SSH research was often not completed as planned. In this context, there continues to be a high level of demand for these grants, and IDG funding in particular fills a unique need for emerging researchers across Canada.

IG/IDG objectives are aligned with federal government priorities, particularly with the “Knowledge” and “People” pillars outlined in Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation 2014. The IG/IDG funding opportunities also directly support SSHRC’s primary strategic outcome for 2013-16, which is “to make Canada a world leader in SSH research and research training.” Providing IG/IDG funding is also aligned with the roles and responsibilities of the federal government when considered alongside the roles and responsibilities of provincial/territorial governments and of industry. The federal government has been funding SSH research for over 50 years, and it is appropriate that it continues to do so in light of the critical importance of research to national economic development, as the world becomes increasingly knowledge based.

Performance

The evaluation concludes that the IG/IDG and their predecessors (SRG/RDI) are contributing to the achievement of their intended immediate and intermediate outcomes, namely the dissemination and potential use research results, strengthened research capacity, and training of HQP that is associated with career benefits. In the longer term, all of these are likely contributing to the recognition of the expertise and excellence of Canada SSH research.

Research excellence: SSHRC funding may be linked to various measures of research excellence, such as Canada’s high scientific impact and high standing in SSH research internationally. Researchers funded through the IG/IDG (and predecessors) have received more prestigious prizes than unfunded researchers.

IG and IDG are both delivered cost-efficiently, particularly in comparison to their respective predecessors, SRG and RDI. SSHRC has implemented many changes to improve efficiency since the last evaluation, but there remains room for other improvements to the program (e.g., application and review process).

Finally, the delivery changes from SRG/RDI to IG/IDG have had mixed results in terms of the reach of these funding opportunities: while IDG have effectively provided access to funding for younger and emerging researchers, success rates have dropped for IG compared to its predecessor SRG as IG project budgets have outpaced available funding. To date, observed impacts of the delivery change on outcomes are small and inconsistent, and a longer timeframe is required to more accurately assess these impacts and recent adjustments made by SSHRC. Overall, IG/IDG appear to be achieving immediate outcomes to a similar extent as their predecessors SRG/RDI, and are on track to achieving long-term outcomes.

Dissemination and use of research results: There is active dissemination of, and broad interest in, the research supported by the IG/IDG funding opportunities (and predecessors), primarily within, but also outside of academia. Compared to unfunded projects, funded projects generally produced more outputs, employed more knowledge dissemination strategies, and were more likely to have their results used. However, there is limited information available on stakeholder use, and even less so on the subsequent benefits of project results. Although SSHRC funding was a key enabler of knowledge mobilization, further efforts to support these activities and recognize their value could help mitigate barriers to the uptake and use of research results that persist within the academic system.

Training: IG/IDG funding also supports training for large numbers of students and other HQP. Funded projects often provide opportunities for networking and the development of a variety of research and professional skills applicable to careers in a variety of areas. These training experiences are associated with positive employment outcomes for students, based on comparisons with employment outcomes of students who did not receive any SSHRC funding. These findings are corroborated by multiple examples, but could be strengthened through the systematic collection of representative information on training experiences and subsequent employment outcomes.

Research capacity: SSHRC funding, particularly through the RDI/IDG, has strengthened research capacity in the majority of funded projects by supporting and enhancing collaborations (established and new), and by enabling the development of new research questions. Those who received RDI/IDG funding are more likely to receive future SSHRC funding, and higher amounts of subsequent funding from SSHRC and other sources.

Recommendations

The evaluation recommends the following for the IG/IDG and for SSHRC:

• Recommendation 1: IG and IDG remain clearly relevant to support SSH research in Canada, are making effective contributions to expected outcomes, and are managed efficiently, and so should continue to receive federal support.
• Recommendation 2: SSHRC should continue to encourage knowledge mobilization of SSH research, including examining the feasibility of improved monitoring of uptake/use of grantee’s research, and through promotion of achievements and best practices.
• Recommendation 3: SSHRC should explore the feasibility of and potential options for following up with trainees hired through IDG and IG in order to collect more robust information on training experiences and career outcomes.
• Recommendation 4: IG/IDG program management should continue to proactively monitor and investigate further the impacts of grant amounts, success rates and teleconference adjudication on the achievement of outcomes.

---

1.0 Introduction

This report presents the results of an evaluation of the Social Sciences and Humanities Research Council’s (SSHRC) Insight Grants (IG) and Insight Development Grants (IDG). The evaluation was conducted to meet the coverage requirements of the Treasury Board’s Policy on Evaluation (2009) and the requirements of the Financial Administration Act.

1.1 Program Description

1.1.1.SSHRC’s Insight Program

SSHRC’s raison d’être is to fund research and research training that “builds knowledge about people, past and present, with a view toward creating a better future ^{Footnote 1}.” SSHRC’s Insight program aims to support and foster excellence in social sciences and humanities (SSH) research that is intended to deepen, widen and increase our collective understanding of individuals and societies, as well as to inform the search for solutions to societal challenges.

The objectives of the Insight program are to:

• build knowledge and understanding from disciplinary, interdisciplinary and/or cross-sector perspectives through support for the best researchers;
• support new approaches to research on complex and important topics, including those that transcend the capacity of any one scholar, institution or discipline;
• provide a high-quality research training experience for students;
• fund research expertise that relates to societal challenges and opportunities; and
• mobilize research knowledge, to and from academic and non-academic audiences, with the potential to lead to intellectual, cultural, social and economic influence, benefit and impact.

The Insight program includes six main funding opportunities, two of which are the focus of the present evaluation: Insight Grants (IG) and Insight Development Grants (IDG).

1.1.2. Insight Grants and Insight Development Grants

Background and objectives

Both IG and IDG were developed as part of SSHRC’s Program Architecture (PA) Renewal. IDG were first offered in 2011-12 and IG were first offered in 2012-13. Prior to the PA renewal, similar funding was available through the Standard Research Grants (SRG) and the Research Development Initiatives (RDI). The key characteristics of the IG and IDG funding opportunities are compared with those of their predecessors (SRG and RDI, respectively) in Appendix A.

Both IG and IDG contribute to the achievement of the outputs and outcomes described in the Insight program logic model (see Appendix B). The focus of IG is on supporting excellence and advancing and mobilizing knowledge in mature research programs, while the focus of IDG is on supporting research in its initial stages to develop research capacity. Both opportunities also seek to provide trainees with research, professional and knowledge mobilization skills that lead to employment in various sectors. In the longer term, IG and IDG are also expected to contribute to national and international recognition of the excellence and expertise of Canadian SSH researchers.

Delivery

Insight Grants provide funds to emerging and established scholars, as individuals or in teams, for long-term research initiatives. Originally, these grants provided up to $500,000 over three to five years, with a minimum of $7,000 requested and a maximum of $200,000 available in a single year. In the October 2015 competition, the maximum value of Insight Grants was reduced to $400,000 over three to five years, with up to $100,000 available per year.

Insight Development Grants support the development of new research questions, and experimentation with new methods, theoretical approaches or ideas. The grants are valued at between $7,000 and $75,000 over one to two years. Funding is allocated across two categories of scholars: established scholars with a record of research achievement, and emerging scholars who have not yet had the opportunity to establish an extensive record of research achievement, but who are in the process of building one.

All grant applications must involve a discipline, thematic area, approach, or subject area eligible for SSHRC funding. SSHRC encourages applicants to submit research proposals that address one or more future challenge areas (i.e., key areas of interest for addressing Canada’s future information needs), but the extent to which the future challenge areas are addressed is not evaluated in the grant proposals. Proposals are reviewed and scored by a merit committee, and recommended proposals are funded within budgets. Once grants are awarded, each grantee must submit an annual financial statement of accounts detailing how funds were used. When grants are completed, each researcher is required to submit a final research report, describing their research program’s impacts and dissemination, outputs, partners, highly qualified personnel (HQP), and success/hindrance factors.

The IG and IDG funding opportunities are administered through SSHRC’s Research Grants and Partnerships Division. In its 2014-15 funding cycle, IG/IDG expenditures (grants and administrative expenditures) were approximately $97 million in total, consisting of $78 million for IG and $19 million for IDG.

1.2. Evaluation Objective and Scope

This evaluation was conducted in accordance with Section 42.1 (1) of the Financial Administration Act and the Treasury Board Policy on Evaluation (2009)^{Footnote 2}, which requires that every department conduct a review every five years of the relevance and effectiveness of each ongoing program for which it is responsible. The IG and IDG predecessors (SRG and RDI) were last evaluated in 2010-11, so the scope of this evaluation primarily covers competition years 2009-10 to 2014-15.

However, as shown in Figure 1, slightly different time periods were examined for each evaluation issue. Questions related to program relevance and efficiency/economy and short-term outcomes focused on the current funding opportunities, namely IG/IDG. To assess longer term outcomes, the evaluation focused on the earlier SRG/RDI funding opportunities. This division of scope was necessary because:

• IDG and IG were first implemented in competition years 2011-12 and 2012-13, respectively. To address a full five-year period, it was necessary to include data on the preceding RDI and SRG.
• Depending on the funding opportunity (IDG are one or two years; IG are three to five years) and individual project, one to five years may pass between the time grants were awarded and completion of a research project. It may then take several more years for research findings to be fully disseminated, for stakeholders to use these findings, and for trainees to move into jobs. Given the relatively short length of time since the initiation of the IG and IDG, these funding opportunities have not yet had sufficient time to contribute to most outcomes. Thus, the SRG/RDI grants spanning back to 2005-06 were examined to assess the achievement of intermediate and long-term outcomes.

Figure 1: Evaluation Scope

Description of figure 1

Figure 1. Evaluation Scope

Figure one is an arrow depicting a linear timeline, by competition year, starting at 2005/06 and ending with 2014/15. The timeline represents how each evaluation issue was examined in each time period. Questions related to program outcomes (both long- and short-term) were used for years 2005/06 to 2010/11. The years from 2011/12 to 2014/15 contain evaluation issues relating to relevance, short-term outcomes (where possible), and efficiency/economy. The timeline also marks that the IDG funding opportunity was initiated in 2011/12 and that the IG funding opportunity was initiated in 2012/13.

1.3. Evaluation Issues and Questions

The evaluation was designed to address the five core evaluation issues stipulated in the Treasury Board Policy on Evaluation (2009), which fall within two broad categories: relevance and performance. This evaluation also addressed specific issues related to the design and delivery of the funding opportunities. Table 1 presents the evaluation questions that were developed to address each evaluation issue.

Table 1 : Evaluation issues and questions

Evaluation issues	Evaluation questions
Relevance
Continued need for the funding opportunities	1. Is there a continued need for IG/IDG in light of the current context?
Alignment with government priorities	2. To what extent are IG/IDG objectives consistent with federal government priorities and SSHRC strategic outcomes?
Alignment with federal roles and responsibilities	3. To what extent is providing IG/IDG funding an appropriate role for the federal government?
Performance (effectiveness, efficiency and economy)
Effectiveness	4. To what extent have the IG/IDG funding opportunities contributed to the expertise and excellence of Canadian SSH researchers being recognized nationally and internationally? 5. To what extent have the IG/IDG funding opportunities contributed to the dissemination and use of research results? 6. To what extent have the IG/IDG funding opportunities contributed to trainees developing skills that lead to employment? 7. To what extent has the IDG funding opportunity contributed to strengthened research capacity?
Demonstration of efficiency and economy	8. To what extent are the IG/IDG funding opportunities delivered in a cost-efficient manner?
Design and delivery	9. What has been the impact of the delivery changes introduced with IG/IDG?

1.4. Evaluation Methods

The evaluation relied on data from multiple lines of evidence to assess the relevance and performance of the IG/IDG funding opportunities. The responsibility for data collection was shared between evaluators from R.A. Malatest and Associates and evaluators from SSHRC’s evaluation division, while evaluators from Science-Metrix drafted this Evaluation Report. Throughout the evaluation process, input was also provided by the Evaluation Advisory Committee, which consisted of program staff and management involved in the delivery of IG/IDG, and representatives from SSHRC’s Corporate Strategy and Performance Division.

The following nine evaluation methods and corresponding data sources were mapped to the evaluation questions and indicators in the evaluation matrix (see Appendix C).

• Document review (n=46)
• Literature review (n=40)
• Administrative data review
  • SRG & RDI final research reports (n=4,116); 2005 to 2009
  • SRG & RDI applications (n=8,749); 2005 to 2011 or 2010
  • IG & IDG applications (n=9,842); 2012 or 2011 to 2014
  • Statement of accounts (summary of trainee stipend payments); 2005 to 2014
• Key informant interviews (n=17)
  • SSHRC management and staff (n=4)
  • SRG/RDI grant recipients (n=4); 2005 to 2010
  • IG/IDG grant recipients (n=5); 2011 to 2014
  • IG/IDG unfunded applicants (n=2); 2011 to 2014
  • Committee members (n=2)
• Student survey (n=6,478 respondents)
  • Students funded through SRG/IG/RDI/IDG stipends (n=1,635)
  • Students funded through SSHRC scholarships/fellowships (n=3,378)
  • Student applicants for SSHRC scholarships/fellowships who were not funded (n=1,465)
• Researcher survey (n=2,933 respondents)
  • SRG/RDI funded researchers (n=1,228); 2005 to 2011 or 2010
  • IG/IDG funded researchers (n=1,033); 2012 or 2011 to 2014
  • SRG/RDI unfunded applicants (n=212); 2005 to 2011 or 2010
  • IG/IDG unfunded applicants (n=460); 2012 or 2011 to 2014
• Case studies (n=9)
  • SRG-funded projects (n=7)
  • RDI-funded projects (n=2)
• Student focus groups (n=61 participants through 9 focus groups)
• Cost-efficiency analysis
  • Financial data on administrative and grant expenditures: SRG and RDI (2005-06 to 2010-11), IG (2013-14 to 2014-15), IDG (2012-2013 to 2014-15)
  • Financial data on comparable funding opportunities: NSERC Discovery Grants (2007-08 to 2014-15)

Initial integration and analysis of the multiple lines of evidence from these methods was conducted by evaluators from SSHRC’s evaluation division, which yielded a first round of key findings. Additional in-depth content analysis was conducted by Science-Metrix to validate these key findings and draft this Evaluation Report. Content analysis of the detailed technical reports and supporting documents was undertaken using ATLAS.ti, a computer-assisted qualitative data analysis software that allowed Science-Metrix to code the evaluation data using both deductive and inductive approaches. This rigorous approach to conducting content analysis ensured that the findings and recommendations are evidence based (i.e., grounded in the available documentation).

Challenges and Mitigation Strategies

Each of the methods used in this study has its strengths and weaknesses. Several of these challenges were identified early in the evaluation process and associated mitigation strategies were thus built into the design (e.g., survey data collected across comparison groups). Data validation steps were also included in the integration and reporting process to compare information collected across multiple sources, and to consider smaller scale limitations specific to individual lines of evidence.

The main remaining challenges relate primarily to the timeframe for this evaluation, especially given the recent implementation of IG/IDG. This and other pertinent challenges are described below, along with the strategies used to address them.

• Recent implementation of IG/IDG: The implementation of the IDG and IG funding opportunities (2011-12 and 2012-13, respectively) are too recent to expect funded research to have achieved most intended outcomes at this time. The evaluation mitigated this by assessing short-term outcomes for these funding opportunities—achievement of which suggests that they may later achieve intermediate and long-term outcomes. The evaluation also addressed this challenge by exploring the extent to which the preceding SRG/RDI funding opportunities contributed to intended outcomes. However, given the differences in their design, it is not possible to conclude that IG/IDG will act in the same manner as their predecessors. This timing also had implications for the assessment of efficiency and design and delivery, as will be discussed in those sections.
• Demonstrating IG/IDG’s contribution to outcomes: Many of the data collection methods used to assess outcomes (e.g., surveys, interviews, focus groups) relied on self-reported data, which are also limited by stakeholder’s memory and perceptions of the funding opportunities’ contribution to outcomes. Furthermore, funded individuals may have been motivated to report positive outcomes. This limits the ability to conclude that IG/IDG were responsible for achieving these outcomes. These limitations were mitigated by using a quasi-experimental design for the researcher and student survey: by comparing grantee/student outcomes to other groups (e.g., unfunded applicants, unfunded students, and students funded through other sources) the evaluation could make stronger claims about the funding opportunities’ contribution to outcomes based on counterfactual arguments, even if conclusions about direct cause and effect could not be made.
• Access to respondents and participation: Primary data collection (e.g., interviews, focus groups, surveys, case studies) relied on SSHRC’s ability to reach respondents and encourage them to participate in the evaluation. This was particularly challenging when it came to student survey and focus group respondents as SSHRC does not have a database of students who are funded through IG/IDG/SRG/RDI grants. Instead, students were reached by contacting grantees and asking them to forward an invitation to students. Only some grantees had up-to-date information for their students and it was not possible to confirm if grantees followed through with the request. More generally, stakeholders are busy individuals and many chose not to respond to participation requests. For the most part, sample sizes were sufficiently large to confidently report findings and respondent characteristics (e.g., demographic information) were compared against the known population in order to assess similarity. This analysis also revealed that survey samples were sufficiently similar so as to make assertions based on comparisons between groups using unweighted data. However, it is possible that respondents may differ in some other meaningful way from those who declined to respond.

---

2.0 Evaluation Findings

This section presents the findings from the evaluation by evaluation question. The main conclusions for each evaluation question are presented at the start of each section, followed by the presentation and discussion of specific findings that support these conclusions.

2.1 Relevance

2.1.1. Is there a continued need for IG/IDG in light of the current context?

Social sciences and humanities (SSH) research contributes to exploring and understanding the human dimension of a broad range of issues of value and interest to Canadians, such as culture, business, education, health, the environment and policy. Recognizing that understanding human thought and behaviour are critical to helping Canada and the world address current and future societal challenges ^{Footnote 3}, numerous key informants and documentary sources clearly portrayed ongoing funding for SSH research as necessary to support Canada’s knowledge base and expertise in these fields ^{Footnote 4}.

Funding through IG/IDG provides Canadian researchers with the means to sustain SSH knowledge and expertise at a high level of quality in order to address critical societal challenges. For example, applications submitted for IG/IDG and their predecessors (SRG/RDI) most commonly sought to further our understanding on questions relating to education, arts and culture, management, politics and government. Of note, research projects examining environment and sustainability issues have become more prevalent in IG/IDG applications compared to SRG applications. IG/IDG-funded research also addresses particular issues of interest, including the six future challenge areas (i.e., higher education, energy and resources, Aboriginal peoples, peak population, emerging technologies, global landscape) that were identified through national and international consultations led by SSHRC. Approximately 20% of IG/IDG applications since 2012 have sought to answer questions related to at least one of these future challenge areas. SSH research that benefits from IG/IDG support therefore has the potential to bring about lasting cultural, social, economic and intellectual impacts on these questions. Specific examples of such impacts will be examined in later sections of this report.

The continued need for SSH research funding through IG/IDG is further highlighted by the sustained high level of demand for these funding opportunities. There are currently about 24,000 full-time SSH professors across Canada ^{Footnote 5}, which help train a growing population of full-time graduate students in SSH disciplines ^{Footnote 6}. A large portion of these researchers submit applications for IG or IDG: on average, from 2011 to 2014, SSHRC received 2,042 applications for IG and 922 applications for IDG each year, with average success rates of 24% and 31%, respectively. Administrative data also show that demand, based on the number of applications, has increased annually since 2011 for IDG. Demand for IG has decreased by about 9% from 2013 to 2015, but remains at nearly 2,000 applications per year.

Ongoing need for SSHRC funding is also confirmed by “repeat business”. Nearly half of surveyed applicants to the IDG and RDI opportunities again applied to SSHRC, whether they had been initially funded or not, suggesting that these researchers need and/or value SSHRC funding. In fact, SSHRC was the most common source from which IDG/RDI applicants sought funding in subsequent applications, ahead of universities, provincial governments and not-for-profit or foundations; each of these groups attracted between 15% and 25% of subsequent applications.

The demand for IDG is particularly strong for emerging researchers, who have submitted 66% of IDG applications (compared to 10% of IG applications) since 2011. This suggests the IDG are meeting a particular need for emerging researchers. Key informants indicated that the IDG funding opportunity was created to fill a clearly-identified gap—i.e., research funding specifically designated for new scholars across Canada—as well as to help established scholars explore new research directions.

This demand is also consistent with the finding that these grants are generally well aligned with the needs of SSH researchers. Both IG and IDG were reported to meet the needs of SSH researchers in supporting their research projects, including for specific types of expenses incurred in the course of their research, such as travel and student support. Approximately 90% of SRG and RDI grantees indicated in their final research reports that SSHRC funding was a contributing factor to the success of their research—the most commonly cited factor, in fact. Other views on the extent to which IG/IDG align with and meet the needs of SSH researchers were largely positive, with most interviewed SSHRC staff and committee members pointing out that these grants were designed to meet the evolving needs of researchers and to accommodate the full range of disciplines and types of research targeted by the grants (e.g., flexibility in the size and duration of grants, ability to recognize excellence in many different forms during the adjudication process).

A key reason that contributes to the continued need and demand observed for IG/IDG is the fact that few alternatives exist in Canada to support SSH research in general, and fewer still in the same manner as the IG/IDG. Based on a review of the other funding opportunities identified in the literature, by key informants and by SSH researchers, funding provided via IG/IDG is unique in terms of eligibility and scope, in that it is accessible to SSH researchers across all regions of Canada, and across an extensive range of topics and research fields. In contrast, some alternative funding opportunities identified were discipline or subject-matter specific or only available to researchers from specific regions/provinces. Grants from other funders were also often more limited in size (i.e., grant amount or duration), or were more restricted in terms of eligible expenses. Notably, there are fewer and less lucrative opportunities for research contracts and private sector funding available to SSH researchers compared to those available to their counterparts in the natural or biomedical sciences ^{Footnote 7}.

Overall, these findings suggest that there would be a clear gap in the absence of these funding opportunities. Indeed, surveyed researchers report that, without requested IG/IDG funding, nearly half of proposed projects did not proceed and less than 16% of projects that proceed were completed as planned. The most common modifications were reduced budget, scope and team, which occurred in at least two thirds of unfunded projects that proceeded without IG/IDG. As will be discussed in later sections of this report, these modifications also led to reduced outcomes, including more limited dissemination of research results and use of these results by stakeholders compared to funded projects.

2.1.2. To what extent are IG/IDG objectives consistent with federal government priorities and SSHRC strategic outcomes?

The IG and IDG objectives are to support research excellence, and to support research in its initial stages, respectively. Projects funded through these grants are expected to contribute to the objectives of the Insight program, which are to build knowledge and understanding, enable the development of new approaches, provide high-quality research training experience for students, fund relevant expertise, and mobilize research knowledge with the potential to lead to intellectual, cultural, social and economic influence, benefit and impact (see detailed objectives in Section 1.1).

As confirmed by key informants, these objectives are clearly aligned with two of the three pillars at the foundation of the federal government’s current science, technology and innovation strategy, Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation 2014. First, supporting research excellence and innovative approaches that build knowledge through the IG/IDG directly contributes to the strategy’s “Knowledge” pillar, which emphasizes support for “research and scientific capacity in universities, colleges and polytechnics” and “for excellence in discovery-driven and applied research ^{Footnote 8}.” This pillar also aims to ensure that “research will become more open, accessible and transparent to the public and end users,” which is supported by the IG/IDG’s knowledge mobilization objectives. Second, the IG/IDG objectives to fund established and emerging SSH researchers as well as student training contribute to the strategy’s “People” pillar: “the Strategy aims to strengthen the skills and capacity that keep Canada at the forefront of research and innovation. This includes […] supporting the researchers who are making ground-breaking discoveries and pushing the frontiers of knowledge ^{Footnote 9}.”

Similarly, by supporting research excellence that builds knowledge and understanding about people and societies, and that enables high-quality research training opportunities, the IG/IDG funding opportunities are highly consistent with SSHRC’s primary strategic outcome for 2013-16, which is “to make Canada a world leader in SSH research and research training ^{Footnote 10}.”

2.1.3. To what extent is providing IG/IDG funding an appropriate role for the federal government?

The federal government has a long history of funding SSH research, going back to the original Standard Research Grants (SRG, IG’s predecessor) funded by the Canada Council in the 1960s. Through SSHRC, the federal government has maintained this role to this day though SSHRC. In 2014, the federal government was the second highest spender in SSH research in Canada (28% of total SSH research spending), after education institutions (52%) ^{Footnote 11}. While provincial governments also support SSH research (11% of total SSH research spending), they play a larger role in providing funding for teaching and post-secondary institutions than for research ^{Footnote 12}. In contrast, the federal government role’s in supporting post-secondary education is primarily focused on research, in accordance with the critical importance of research to national economic development—a key area of federal responsibility.

This last point was emphasized by key informants, who agreed that it was appropriate for the federal government to fund SSH research through the IG and IDG because such investments support and inform the research and development activities that drive an innovation-based economy. In other words, federal funding for SSH research is necessary to ensure that Canada remains competitive as the world becomes increasingly knowledge based.

Research and development in the natural sciences and engineering (NSE) attracts a large proportion of its funding from the business sector (51% of total NSE research spending; $27.8 billion in 2014) ^{Footnote 13}. Because industry expenditures on SSH research and development are much less extensive, a large part of the responsibility for supporting this work is seen to rest on the federal government. That said, federal government funders often require projects to attract matching funds from other sponsors or partners, in which case federal grants “[become] a catalyst for investments from other sectors ^{Footnote 14}.” Incidentally, as the largest customer for SSH contract research in Canada ^{Footnote 15}, the federal government also appears to be one of the main users of SSH research.

2.2 Performance

The next four evaluation questions examine the achievement of expected outcomes. Findings for these questions cover the IG/IDG funding opportunities, but will focus on grants funded through their predecessors from 2005 onward: Standard Research Grants (SRG) were offered until the IG was implemented in 2012, while the Research Development Initiatives (RDI) were offered until the IDG was implemented in 2011. This broad scope is necessary to capture intermediate and longer-term outcomes that are expected in the years after the completion of research projects, namely uptake and use of the results, strengthened research capacity, employment outcomes for students, and recognition of research excellence. At the time of this evaluation, it was too early for the IG/IDG funding opportunities to have contributed to most of these outcomes.

To provide a larger data sample, results from the SRG and IG were grouped in some analyses, as were those from the RDI and IDG. Given differences between IG/IDG and their previous iterations, interpretation of these analyses should consider that IG and IDG outcomes may differ over time. However, as will be examined in Section 2.2.6, impacts on outcomes of the delivery changes introduced with IG/IDG appear to be small and inconsistent.

2.2.1. To what extent have the IG/IDG funding opportunities contributed to the expertise and excellence of Canadian SSH researchers being recognized?

Research excellence can be defined and measured in different ways, such as through the quality of research methods and publications, the effectiveness of knowledge mobilization and uptake, or the subsequent career outcomes for both researchers and trainees. This section focuses on indicators that reflect research excellence—specifically prizes, scientific impact, and university rankings—that are not covered in previous sections and that help assess the achievement one of the longer term outcomes of the Insight program: that the excellence and expertise of Canadian SSH researchers is recognized on the national and international level. This will be followed by a brief discussion of factors that facilitate or hinder research excellence.

Recognition and prizes

The researcher survey revealed a few small but significant differences between researchers with funded or unfunded projects whose body of work has been recognized in the form of prizes or awards since their SRG/IG/RDI/IDG application. A very small number of SRG/IG-funded researchers received the most prestigious prizes available to SSH researchers (4%), while none of the unfunded researchers received such prizes. SRG/IG-funded researchers were also more likely to receive Canadian prizes (12%) and other prizes and recognition (17%) than unfunded researchers (3% and 9%, respectively). The low frequency reported by researchers is not surprising, given that the prizes listed in the survey are rare and often very prestigious (e.g., Nobel Prize, Holberg International Memorial Prize, John W. Kluge Prize for Achievement in the Study of Humanity) ^{Footnote 16}. No significant differences in recognition were found between researchers with funded and unfunded RDI/IDG projects; this is to be expected since researchers receiving RDI/IDG funding were less likely to have established careers at the time than those receiving SRG/IG funding, especially emerging researchers applying for IDG.

Project-specific research contributions, such as books and articles, have also been recognized by Canadian and international organizations through prizes, or other awards, distinction or honourable mentions. At the time of final research report submission, approximately 6% of SRG/RDI funded projects had received official national or international recognition related to their research contributions, and some projects had received multiple prizes. The case studies provided further examples of recognition for research excellence for researchers funded through the IG/IDG (and predecessors), such honorary doctorates, guest editorship of research journals, appointments to positions in national or international academic associations, and nominations to the Royal Society of Canada.

Scientific impact and university rankings

As will be discussed in Section 2.2.2, projects funded through the IG/IDG (and predecessors) produced more research outputs, employed more knowledge dissemination strategies, and were more likely to have their results used by stakeholders. To complement these findings, Canada’s scientific impact was compared to that of other countries with the highest numbers of SSH publications using two indicators: the Average of Relative Citations (ARC) and the Average of Relative Impact Factors (ARIF). The ARC is an indicator of scientific impact based on the number of citations received by peer-reviewed papers, and the ARIF is an indicator of scientific quality based on the impact factor of journals in which these papers are published. Both indicators are “relative” because they account for the different citation patterns in different research fields. These indicators consider all Canadian SSH publications (albeit primarily in the social sciences), not only those supported by the IG/IDG (and predecessors), but it is reasonable to expect that SRG/IG may have contributed to these publications as they are the main funding opportunities for SSH research in Canada ^{Footnote 17}.

Canada is among the five countries that published the most English-language SSH publications from 2005-2013, as indexed in Thomson Reuters’ Web of Science. Among these countries, Canada’s ARC is third highest and ARIF is second highest (Figure 2 ).

Figure 2: Average of Relative Impact Factors of Countries with the Most SSH Publications, 2005-2013

Description of figure 2

Figure 2. Average of Relative Impact Factors of Countries with the Most SSH Publications, 2005-2013

This line graph represents the number of English-language social sciences and humanities publications produced in the five countries that published the most of these. The y-axis represents each of the five country’s average of relative impact factors, beginning at 0.85 and increasing by increments of 0.05, up to 1.20. The x-axis shows each year from 2005 to 2013. The five countries represented on the graph, in order of highest to lowest average of relative impact factors, are the United States, which starts at 1.16 in 2015 and goes to 1.15 in 2013; Canada, which goes from 1.05 to 1.12; the United Kingdom, which goes from 1.00 to 1.04; Germany, which goes from 0.89 to 0.96; and Australia, which goes from 0.90 to 0.97. Each country shows a fairly steady incline overall, with the United States and the United Kingdom each dropping 0.05 between the years 2012 and 2013.

Overall, Canada’s ARC and ARIF are also slightly higher than the combined scores of 23 other countries, with a similar number of English language SSH publications (data not shown). This means that Canada’s SSH publications, on average, are more frequently cited and are published in higher impact journals than publications from most other countries, and suggests that Canada’s SSH publications are competitive on the world stage.

With regard to university rankings, there appears to be a relationship between SRG funding and the position of Canadian universities in international rankings. Three Canadian universities—Toronto, British Columbia, McGill—consistently ranked in the top 50 universities worldwide (2010-2014) and these same universities received the highest amounts of SRG funding leading up to these rankings. Although these data do not account for other factors that may play a role in university rankings (or university funding amounts) and cannot show causality, this positive association is consistent with a scenario in which IG/IDG funding (and predecessors) may have helped to increase the prominence and recognition of these universities internationally.

As shown throughout this section, Canadian excellence in SSH research is recognized internationally. Adding to the examples presented previously, the summaries from the case studies in Table 2 illustrate how IG/IDG-funded research can be associated with various measures of scientific excellence.

Table 2: Examples of research excellence

An SRG-project led by Sherry Stewart demonstrated that specific personality factors may result in individuals engaging in drinking behaviours for different motives (e.g., coping motives). These findings have been applied by clinicians, front-line staff (e.g., counsellors on the university campus), clinical psychology graduate students, and researchers working in the field of addictions. These practitioners can now better recognize and explore motivations for alcohol use in the clients they treat and thus offer targeted prevention and therapeutic interventions adapted to their clients’ individual motives. This study also influenced Dr. Stewart’s future research efforts, whose findings are currently being used in several European countries, Canada and Australia. Two analyses of research productivity in clinical psychology professors confirm that Dr. Stewart is the most productive female psychologist in Canada and the most or second most (depending on the analysis) productive clinical psychology researcher in Canada overall. As shown by her active involvement with alcohol research associations and stakeholder organizations, Dr. Stewart’s work has greatly contributed to an improved understanding of mental health and addictions, both nationally and internationally.

With SRG funding, Serge Lusignan presented a new social history of the picard language in the Middle Ages that provides a comprehensive description of social and political conditions that enabled the use of this language to expand between the 8th and 15th centuries. The project bridged two disciplines that study language in different ways—history and linguistics—and Dr. Lusignan shared the method he used to do so in his book on the picard language. Other results of the project include over a dozen papers and almost 30 presentations. Dr. Lusignan has long been a globally recognized scholar and this work led to further recognition from leading international institutions (e.g., invited presentations), as well as uptake in further research and teaching by several researchers in related fields.

Success and hindrance factors for research excellence

Many of the same success factors identified in later sections—those which facilitate knowledge mobilization, capacity building, and effective training—also directly contribute to research excellence. This includes research funding from SSHRC and other sources, particularly flexible and/or consistent funding that is allocated through a stringent review process. Funding is perceived to be most beneficial when it supports a comprehensive and rigorous program of research and associated knowledge mobilization activities, and helps researchers attract motivated and qualified students to participate in their projects.

Indeed, about 60% of SRG and RDI recipients identified access to qualified students as one of the main success factors of their projects, a view further emphasized by case study participants. Conversely, challenges with student recruitment were seen to be a hindering factor by about 20% of grantees. The ability to engage and build relationships with collaborators and partners, which requires both funding and time, was also often cited as a success factor by several grantees and case study participants. On the other hand, the lack of time for researchers to devote themselves to research activities was cited by nearly a quarter of SRG/RDI grantees as a hindrance factor.

Another success factor identified by about one third of recipients was the availability and/or access to good quality data (e.g., reliable, consistent, sustained participant recruitment); this was also seen as a hindrance in about 20% of projects when such data were not available. As an example from the case studies that illustrates both sides of this, one project greatly benefited from access to census data from Statistics Canada until 2011, after which the abolition of the mandatory long form had a negative impact on the quality of the available data and the ability for researchers to analyse trends over time.

Finally, almost no challenges were raised across the different lines of evidence that might be faced by interdisciplinary research in the IG/IDG funding opportunities, even if the literature indicated this type of work could be at a disadvantage ^{Footnote 18}. This issue was the object of a recommendation in the previous evaluation of the SRG program (i.e., to establish additional inter- and multidisciplinary review committee, which has been in place for the IG since 2012). The lack of concerns observed in this evaluation suggests SSHRC’s actions have been effective to reduce this challenge for the IG/IDG.

2.2.2. To what extent have the IG/IDG funding opportunities contributed to the dissemination and use of research results?

Production and dissemination of research results

Almost every single grant recipient produced research results across all the funding opportunities examined. Analysis of final research reports indicates that 95% of RDI and 98% of SRG projects generated at least one research output by the end of the grant, and many generated multiple types of outputs. The most common research contributions identified across multiple lines of evidence (including the researcher survey and the case studies) included conference presentations, journal articles, course content, websites and other web-based content. Books and book chapters were also common outputs for SRG/IG, as these are the preferred dissemination mechanism in some SSH disciplines. SSHRC funding was also associated with a greater number of total research outputs. For both completed and ongoing SRG/IG projects, surveyed researchers reported that significantly more outputs were produced (means of 20.6 vs. 10.9) and planned (means of 11.9 vs. 8.8) for funded projects compared to unfunded projects. A similar difference was observed between funded and unfunded RDI/IDG projects (means 9.6 vs. 7.3), but this difference was not statistically significant.

These research results were disseminated using a range of knowledge mobilization strategies, with most grant recipients targeting at least one type of audience. Again, SSHRC funding was associated with knowledge dissemination: according to the researcher survey, more dissemination strategies were used (means of 4.8 vs. 3.9) and planned (means of 4.1 vs. 3.9) for funded SRG/IG projects compared to unfunded projects. A smaller, non-significant difference was observed between the number of dissemination strategies reported for funded and unfunded RDI/IDG projects.

Not surprisingly, given the most common types of outputs noted above (e.g., publications, conferences), almost all funded researchers used strategies to target other academics. The vast majority (89-97%) of grant recipients sought to share their findings with academics within their own field of research and, to a lesser extent (for about two-thirds of respondents) with academics in other disciplines. Dissemination to non-academic audiences such as decision-makers in the public, private and not-for-profit sectors was less frequent, but was still reported by nearly half of SRG and RDI grant recipients in their final research reports. Moreover, about of half of surveyed researchers reported having already written or presented findings to a wider public, and up to two thirds planned to do so.

Evidence from the case studies and interviews confirms that researchers implemented a broader range of knowledge mobilization activities to target non-academic audiences than for academic audiences. Knowledge mobilization, as defined by SSHRC, encompasses a wide range of activities to enable the “reciprocal and complementary flow and uptake of research knowledge between researchers, knowledge brokers and knowledge users—both within and beyond academia—in such a way that may benefit users and create positive impacts within Canada and/or internationally” ^{Footnote 19}. Knowledge mobilization strategies used by SRG/RDI recipients include the development of learning materials for schools (from elementary to post-secondary), cross-sectoral symposia (e.g., gathering scholars and artists), presentations to practitioners and clinicians, reports prepared for government departments or NGOs, opinion editorials, public lectures, and social media outreach.

The case studies also showed that the contact between grant recipients and other stakeholders takes many forms, with the latter sometimes acting either as active partners in knowledge mobilization or as potential users of research results—or as both. For example, researchers were invited to share their project findings by partners in the academic community, by government bodies (including appearances before parliamentary committees), by associations and groups that inform policies and programs (e.g., think tanks), by industry (e.g., advisory services), and by media outlets. Consequently, research results made possible by SSHRC funding have been shared more broadly with researchers and students, policy-makers at various levels (e.g., municipal, provincial, federal, international), practitioners (e.g., in health or education, judges, artists), First Nations communities, industry, schools, and the broader public.

Not every grant recipient has used every type of knowledge mobilization approach or reached every type of audience, nor would it be appropriate for them to do so. This makes it difficult to assess the extent to which IG/IDG funded research is being disseminated as expected, as quantitative indicators are inadequate to assess when or to what extent specific strategies to mobilize knowledge to a given audience are appropriate or beneficial, particularly in the absence of a baseline against which to compare. Because of the variety of knowledge mobilization strategies that are used and the range of stakeholders that are reached, it particularly challenging to capture and measure the frequency and effectiveness of knowledge mobilization to reach non-academic audiences, compared to the more traditional strategies used to reach academic audiences. Despite these challenges, these findings point to active dissemination of, and a broad interest in, the research supported by the IG/IDG funding opportunities (and their predecessors), primarily within, but also outside of academia.

Use and benefits of research results

The active dissemination of research results points to the potential for uptake and use of these results by stakeholders, and for subsequent benefits to occur (e.g., intellectual, cultural, social and economic influence and impact). However, limited information was available on the actual use of research results stemming from IG/IDG and SRG/RDI funded research, particularly in non-academic contexts, and even less so on the subsequent benefits of such use.

Evidence from interviews, surveys and research reports was limited by the perceptions of researchers, who were often unable or hesitant to comment on the non-academic use or benefits of their work. These limitations were also associated with timing and attribution: both interviewees and the literature ^{Footnote 20} highlighted the fact that impacts of SSH research often occur over the long term—generally longer than in other fields (e.g., NSE)—and may be indirect. For example, new knowledge and understanding on issues relevant to policy may not immediately or directly result in changes in policy, but still inform and influence the thinking of policy-makers, or could even lead to them deciding not to make any changes to existing policy. In the case of IG and IDG specifically, many funded projects were not yet completed, making the observation of intellectual, cultural, social and economic benefits premature. The following findings are therefore based on available, albeit limited, evidence: namely, the perceived and expected uptake and use of project results as reported by researchers, as well as anecdotal evidence of benefits from a broader range of sources (e.g., interviews with research users, documentary evidence) consulted as part of the case studies.

Based on the researcher survey, IG/IDG funding (and predecessors) is often associated with the anticipated use of research results by other stakeholders. Overall, funded SRG/IG projects were more likely than unfunded projects to have their results used by any type of stakeholder (89% vs. 78%); a similar difference for RDI/IDG projects (82% funded vs. 74% unfunded) was not statistically significant. Again, other academics (82% for SRG/IG, 69% for RDI/IDG) were by far the most common stakeholder type that was expected to use project findings, such as to further generate or expand knowledge. About one third of funded researchers also reported intended use by not-for-profits (e.g., charitable organizations, NGOs, foundations), while about 20% reported intended use by government stakeholders. Of note, more unfunded projects produced results with intended use by industry (12-15% unfunded vs. 9% funded), a difference that was statistically significant for RDI/IDG projects.

Impacts of completed projects were primarily seen or expected by funded researchers to occur in areas relating to culture, in social spheres, and in the development of policies and programs. For example, in the final research reports, approximately half of SRG and RDI grantees were confident that their research findings would have an impact on teaching and professional practice and on the understanding of social issues and development, whereas 40% of SRG grantees expected impacts on the understanding of culture. Then again, about a quarter of SRG and RDI grantees reported that impacts on an understanding of culture and/or an understanding of economic development, policies or practices were not applicable to their funded project.

More specific examples of use and benefits were observed in the case studies and interviews, but these remain anecdotal since case studies are not representative of all funded projects; this evidence was also quite labour-intensive to collect. Use of research results by other scholars, including some in other countries, was often observed, including the adoption of new methods/approaches developed as part of funded projects. Key informants pointed to IG-funded research findings on male femininity that have been taken up in a clinical setting and incorporated into textbooks on human sexuality, evolutionary psychology, and anthropology; meanwhile, research on fragments of the Berlin Wall supported by another IG project has been featured in exhibits at public art galleries. Other notable examples from projects funded in the last ten years are presented in Table 3.

Table 3: Examples of dissemination and use of research results

Cheryl Suzack’s project “Women’s Writing, Case Law, and Indigenous Feminism” helped show how Indigenous women’s writing reflected and countered representations of Indigenous women and social justice issues in the law and in government policy. Among other uses of her research, through the Canadian Institute for the Administration of Justice, Dr. Suzack has applied the knowledge gained through her SRG-funded research to provide training and guidance to judges writing legal decisions in order to promote more culturally appropriate, easily accessible and understood written judgements. In one case, this guidance has been ongoing informally over several years and has included approximately 250 prepared judgements to date. The legal decisions her work has influenced can also be accessed and used by litigants representing themselves in court, becoming tools with even broader benefits.

Various organizations in the public sector have benefited from new knowledge and theory on the spatial dynamics of national economies resulting from an SRG-funded project led by Mario Polèse. Municipal governments (e.g., Montreal), the Atlantic Canada Opportunities Agency, and Canada Economic Development have reached out to Dr. Polèse with questions about this research. The latter has used the research to develop an indicator to identify vulnerable regions and thus inform decisions on which regions should be prioritized for financial aid.

SRG-funded research led by Nancy Turner has contributed to the cultural revitalization of the communities involved in the project “Dynamics of traditional ecological knowledge acquisition and transmission project.” In collaboration with students of Hartley Bay’s Gitga’at community (British Columbia), Dr. Turner wrote a book containing pictures, names (in English, Latin and Gitga’at), and uses of traditional local plants. Copies of the book were distributed to all community members in Hartley Bay and feedback has been overwhelmingly positive; the community and the book’s contributors have expressed great pride about this book. Nearly a decade after its distribution, the Gitga’at plant book is still being incorporated into the curriculum by local schoolteachers and is helping children learn about local plants and their traditional uses. This book, along with other collaborations with Dr. Turner, are valued in the Gitga’at community for helping them learn and preserve knowledge about themselves. A collaborator noted: “Dr. Turner has reminded us of how important we are.”

Craig Forcese’s SRG-funded project “Preventive Detention in the Securitized State” has resulted in ideas that have influenced both law enforcement and policy-makers. For example, representatives from RCMP have praised Forcese’s work for the practicality and immediate usability of his research, which has heighted the RCMP’s awareness of preventive detention as a tool, as well as the great discretion required when employing it: “if you ignore this research, you run some risks… charter risks, risks of litigation, risks of imprisoning people wrongly, risks of infringing on people’s human rights and freedoms.” The RCMP has also used some of Forcese’s ideas in the development of training materials for investigators, such as schematics and hypothetical situations.

Success and hindrance factors for knowledge mobilization and use

SSHRC funding was often cited by grantees as the main factor that contributed to knowledge mobilization and use of research results, including by supporting travel (e.g., to research sites, to conferences), networking (e.g., hosting symposia), equipment and expertise (e.g., to hire students or communication specialists), as well as costs for open access publication. The fact that SRG/IG-funded projects generally produced more outputs, employed more knowledge mobilization strategies, and were more likely to have their results used suggests that grant recipients were appropriately selected (i.e., researchers with the greatest potential to generate and disseminate knowledge) and/or that funding helped researchers to produce, disseminate and use their research results; these two options are not mutually exclusive. The fact that differences in these outcomes between funded and unfunded RDI/IDG projects are less pronounced and not statistically significant is more difficult to explain. It may have to do with the stage of funded research or researchers (i.e., emerging), with the small sample size that reduced the strength of statistical analyses, or with aspects of the RDI and IDG funding opportunities (e.g., objectives, selection process, size, duration) that reduce the difference between funded and unfunded projects.

Some tensions were also noted in the literature ^{Footnote 21}, case studies and by interviewees between activities and knowledge mobilization strategies that are valued and appropriate in academic and non-academic settings. In short, academic systems tend to encourage and prioritize the dissemination through scholarly publications for academic audiences, whereas non-academic stakeholders often require different types and a broader range of knowledge mobilization strategies that make knowledge more tangible, more broadly accessible, or more directly applicable for these stakeholders. Effective knowledge mobilization is also enabled by positive working relationships between researchers and multiple groups (e.g., collaborators, research participants, research users, communities and stakeholders). Such relationships are based on trust and mutual respect, and take time, effort and resources to develop and maintain (e.g., through networks and face-to-face interactions).

Consequently, further efforts to support and recognize the value of knowledge mobilization activities may help mitigate barriers to the uptake and use of research results that persist within the academic system. For example, some interviewees and case study participants suggested that research funders and institutions could provide support that is specifically focused on knowledge mobilization and/or support knowledge mobilization specialists at universities. Such suggestions were similar to practices encouraged by a group at York University who has been developing institutional capacity to improve knowledge mobilization ^{Footnote 22}.

2.2.3. To what extent has the IDG funding opportunity contributed to strengthened research capacity?

By supporting research in its initial stages, the RDI/IDG funding opportunities are expected to develop and strengthen research capacity within Canadian postsecondary institutions. The RDI sought to achieve this outcome by supporting the development of new ways of analyzing, structuring, integrating and transferring knowledge. Subsequently, the IDG was designed not only to enable the development of new research questions, methods or theoretical approaches, but also includes an explicit goal of enabling early-career researchers to become more competitive in other grant competitions. As such, a portion of IDG funds (e.g., 50% in 2015) is reserved for applications from emerging researchers. While SRG/IG funding also enhances research capacity within postsecondary institutions, these grants are not explicitly intended to support early-career researchers. As such, outcomes from these funding opportunities are discussed more briefly than for RDI/IDG under this evaluation question, and only when relevant.

Collaborations

Developing and maintaining collaborations and partnerships contribute to strengthening research capacity by creating mutually beneficial opportunities for innovation and by enhancing knowledge production, mobilization and training. Collaborations enabled by the IG/IDG (and predecessors) can help build knowledge and develop approaches from interdisciplinary, cross-sector and international perspectives, thus contributing to enhanced individual and institutional capacity for future research. The most common types of collaborators identified by SSHRC grantees are other researchers within the grantees’ institution or at other institutions, followed by collaborators in non-profit organizations, government and industry.

As reported by RDI/IDG applicants, funded projects involved more collaborations overall than unfunded projects (means of 1.92 vs. 1.36). However, the number of new collaborations specifically did not significantly differ between funded and unfunded projects, nor did the proportions of projects with new collaborations (66% of funded projects vs. 72% unfunded). Even so, more than three quarters of RDI/IDG recipients indicated that they would have been unlikely to develop their new collaborations without SSRHC funding. Collectively, these findings suggests that while RDI/IDG funding clearly supports collaborations (both established and new), it does not necessarily lead the development of more collaborations. A similar finding applies to formal partnerships: 31% of RDI/IDG grantees report having developed partnerships as part of their project, about the same proportion as unfunded applicants (26%). Key informant interviewees perceived this outcome as being in line with expectations given the relatively short duration and modest grant size of the RDI/IDG funding opportunities.

Several differences were also observed in the nature and level of involvement of collaborators during the project based on funding status. These differences suggest that SSHRC funding was leveraged to increase and broaden the involvement and contribution of collaborators.

• The level of involvement during the project itself was more likely to occur as planned or to increase compared to initial plans in the case of funded projects, whereas the level of involvement more often decreased or shifted to different aspects in unfunded projects. This difference was observed for both RDI/IDG and SRG/IG funding opportunities.
• SRG/IG recipients were more likely to involve collaborators in data collection, analysis, and knowledge dissemination than unfunded applicants. Similar differences were observed for RDI/IDG, but were not statistically significant.
• Approximately half of RDI/IDG and SRG/IG grantees reported having built research teams or alliances that generated new knowledge based on ongoing knowledge exchange, compared to about 40% of unfunded applicants.
• In their final research reports, 41% of researchers with funded RDI projects and 45% of researchers with funded SRG projects indicated they definitely expected their projects to have an impact on international collaborations. Case study examples also showed how SRG/RDI grants enabled international collaborations that have enriched the scope and reach of their projects, and strengthened the research capacity of their teams, including students.

Also relevant from a research capacity perspective is the finding that RDI/IDG applicants were more likely to expect their new collaborations would continue or intensify (65%) after the end of their funded project compared to unfunded projects (49%). More data would be needed to confirm if RDI/IDG funding have supported the development of long-term collaborative relationships that persist after the end of the projects. On an anecdotal basis, the two case studies illustrated how RDI projects have led to ongoing connections with collaborators and partners in various sectors (e.g., academic, government, practitioners, First Nations), nationally and internationally, which have provided opportunities for further research, student training, networking, formal partnerships, knowledge mobilization, and potential policy impacts.

New questions, methods, tools and theoretical approaches

Enhancing research capacity also takes the form of new questions, methods, tools and theoretical approaches that can be applied in future research. Evidence from final research reports, surveys and case studies suggests that SSHRC funding has strengthened research capacity for RDI/IDG recipients, particularly through the development of new research questions. As shown in Table 4, a large majority of RDI/IDG recipients have developed new research questions and were more likely to do so than unfunded applicants. SSHRC funding also contributes to strengthening research capacity by enabling the involvement of trainees: students funded through RDI/IDG and SRG/IG grants were about twice as likely to have developed new theories, new research methods, and new practices than students who were funded through SSHRC scholarships/fellowships, or not funded by SSHRC, as captured in the student survey.

Table 4: Percentage of RDI/IDG respondents indicating new research questions, tools, methods, or theoretical approaches were developed for their project, disaggregated by funding status

Funding group	New research questions**	New theoretical approaches	New research tools	New research methods*
Funded	94%	61%	49%	44%
Unfunded	86%	62%	45%	53%

* p < .05; ** p < .01; *** p < .001
Source: Survey of SRG/IG and RDI/IDG applicants

About 60% of RDI/IDG applicants developed new theoretical approaches, regardless of whether or not their application was funded (Table 4 ). Some case studies illustrated how SRG-funded researchers have been applying new theoretical approaches in subsequent projects. In one notable example, a three-pronged approach developed to understand drinking motivations in undergraduate students was later applied to examine these motivations in high school students, and adapted to study gambling motivations. The case studies even provided examples of how the funding had allowed SRG/RDI funding recipients to develop relatively new research fields (e.g., history of children and youth, Indigenous feminism). One researcher emphasized that SSHRC had provided visibility, credibility and legitimacy to a new field of study by supporting the development of theoretical approaches in this field through SRG funding.

About half of RDI/IDG applicants also reported having developed new research tools, regardless of whether they were funded through these opportunities, while 38% of RDI recipients were confident that their research findings would contribute to the enhancement of “research infrastructure” (e.g., databases, networks, collections). Meanwhile, slightly less than half of RDI/IDG grantees reported that they had developed new research methods (44%), a proportion similar to that observed in the final research reports of RDI recipients (47%). However, based on the survey data (Table 4 ), unfunded RDI/IDG applicants were more likely to develop new research methods (53%) than funded applicants. It is not possible to speculate as to possible reasons for this difference because the survey data spans both the RDI and IDG, and multiple variables that may differ between these two funding opportunities (e.g., types of projects/scholars funded, differences in implementation) could be affecting this result.

Overall, significant majorities of respondents with funded RDI/IDG projects (73-84%) indicated the development of new research questions, tools, methods, and theoretical approaches would have been unlikely without SSHRC funding. As an RDI recipient explained, such grants are essential for testing out basic hypotheses and methods to determine which approaches work best, and which do not merit to be pursued in future work: “there are certain things I learned from it I would not do again.” Some interviewees also described the IDG funding opportunity as “foundational”, in that it helps researchers to develop a strong base upon which to expand beyond initial questions to a point where it can be supported with other larger grants. The literature also points to the value of small-scale projects in “kickstarting” research careers, leading to larger projects and important results, assuming that these grants are not so small as to negatively constrain the work and that funding from other sources can be found to ensure continuity of the research ^{Footnote 23}.

Subsequent funding

It is widely recognized in the literature that in research, funding begets funding ^{Footnote 24}. This may be due to the recognition or credibility conferred by previous grants, and result from foundational work that was accomplished with previous grants, as explained above. In the case of researchers who successfully applied for RDI/IDG funding, their success rates for subsequent SSHRC applications (in any of 10 funding opportunities) were significantly higher than those of unfunded applicants: 37% vs. 21% for RDI; 34% vs. 20% for IDG. Moreover, of those who received other funding, RDI/IDG grantees received about double the amount of research funding on average per year than unfunded applicants (means of $61,210 vs. $31,415), even if unfunded applicants reported receiving other funding sooner than RDI/IDG grantees. More than half (58%) received other funding while still receiving their RDI/IDG funding.^{Footnote 25}

These findings as a whole suggest that RDI/IDG recipients may be more competitive when applying for subsequent funding, resulting in higher success rates and larger grants. The case studies also provided examples of how RDI projects have not only inspired work that was funded by subsequent larger grants (e.g., SSHRC’s Partnership Development Grant), but also influenced the work of collaborators, who have since been awarded their own SSHRC grants for follow-up projects. As such, RDI/IDG grants have quite possibly strengthened research capacity beyond what has been reported by recipients alone.

2.2.4. To what extent have the IG/IDG funding opportunities contributed to trainees developing skills that lead to employment?

As mentioned above, providing opportunities for students and other HQP to participate in collaborative research and develop new research questions, methods and theoretical approaches contributes to strengthened research capacity. But supporting the involvement of trainees is also expected to enable them to gain experience and develop skills that can lead to subsequent employment in academia and other sectors. This is a key expected outcome of the IG/IDG funding opportunities, and so merited particular attention in this evaluation. The following findings draw on data collected through several methods, two of which were specific to this questionstudent focus groups and the student survey.

To capture evidence of longer term impacts of training, both the focus groups and student survey reached trainees from across Canada who had been funded (i.e., received stipends or salaries) for their participation in SRG/RDI and IG/IDG grants from 2005 to 2013. The survey sample also included students who had received SSHRC graduate-level scholarships and fellowships, and applicants for SSHRC scholarships/fellowships who were not funded. This allowed for comparisons between these groups and those funded though IG/IDG grants (and predecessors), as appropriate (e.g., considering their academic level). In this section, the terms “IG/IDG funding” or “IG/IDG-funded” include funding and opportunities offered through the SRG/RDI unless otherwise specified.

Trainee involvement and skill development

One of the goals of IG/IDG grants is to provide training opportunities for students to develop a variety of skills through hands-on experience. To support this goal, students and other HQP can be hired and compensated using grant funds to work on a variety of project-related tasks. SSHRC grants are clearly being used for this purpose: on average, from 2005 to 2014, 22-34% of grant funds from SRG, RDI, IG and IDG were dedicated to trainee salaries and stipends (calculated from SSHRC’s administrative data); note that these calculations do not include grant funds used to support student travel for project-related activities, such as data collection or conferences. The SRG and RDI final research reports further indicate that 98% of completed projects involved hiring of HQP, and almost all projects involved hiring students (97%). On average, SRG projects hired 6.5 students, while RDIs hired 4. In addition, approximately 40% of SRG and RDI recipients hired non-students with their grant. Hiring of non-students was also more frequent in projects with higher grant values. The most common types of individuals hired were Canadian graduate students, with 70% of RDI and SRG projects involving at least one Masters-level student, and 70% of SRG and 60% of RDI projects involving at least one Doctoral-level student. Nearly 25% of SRG projects also hired at least one foreign Doctoral-level student.

The diversity of activities undertaken by students, coupled with the degree of freedom and responsibilities offered to them, suggests that IG/IDG grants provide extensive and rich opportunities to gain SSH research experience. Most focus groups participants described an experiential learning context, often through a combination of hands-on learning and guidance from the lead researcher and/or other team members (e.g., discussions of techniques and approaches, suggestions for improvement). Research activities and skills most frequently identified by students in the focus groups, case studies and survey include data collection associated with quantitative and qualitative methods (e.g., archival research, literature reviews, surveys, focus groups, interviews), and a wide range of discipline-specific techniques. In fact, slightly more IG/IDG-funded students reported involvement in data collection and research implementation than did students with SSHRC scholarships/fellowships or those that were not funded by SSHRC.

A large majority of surveyed students also reported a high degree of involvement in other research-related activities (e.g., development of research ideas/questions, interpretation of findings), whether they were funded through the IG/IDG or other sources. On one hand, unfunded students were more often involved in the development of research ideas/questions than IG/IDG-funded students, but the latter were more involved in knowledge mobilization and research integrity/ethical conduct activities. IG/IDG-funded students also tended to produce more articles, on average, than students who were not funded by SSHRC and were more likely to have their findings cited by others than either unfunded students or those who had received SSHRC scholarships and fellowships. In the words of one student: “Articles in journals require a certain level of quality, so my participation in this project not only allowed me but required me to produce work of a superior quality.”

Most surveyed students also had at least some involvement in teaching activities (e.g., presentations, teaching a university course, developing course materials), and in personal and professional-related activities, especially critical or creative thinking, interpersonal communication, teamwork, networking and collaboration, and strategic planning and advice. Of these, networking opportunities, such as through collaboration and participation in conferences, were often highlighted by students as being useful, whether by being exposed to research being done by others, or by meeting key researchers and practitioners in the field. Notably, IG/IDG-funded students tended to produce more oral and poster conference presentations at international venues than students who were not funded by SSHRC.

Students generally had positive opinions of the training they received with the support of the IG/IDG. For example, students funded through these grants tended to be more satisfied with their overall research environment (e.g., supervision provided, equipment, infrastructure) when compared to the other two main comparison groups. Moreover, consistently higher proportions of IG/IDG-funded students were satisfied with the personal/professional (67%), teaching (66%) and research skills (85%) than the students who had not received any SSHRC funding (53-70%). When compared to the SSHRC scholarships and fellowships recipients, students hired through IG/IDG were more satisfied with the length of the training (and funding) they received, but were less satisfied with the amount of money they received.

As the previous findings suggest, however, there has been some variability in the perceived quality of the training, with low proportion of IG/IDG-funded students expressing dissatisfaction with the extent to which they acquired teaching skills (8%) and personal/professional skills (6%); only 3% expressed dissatisfaction with the research skills acquired. IG/IDG-funded students also reported less involvement in some types of activities that SSHRC scholarships and fellowships recipients and unfunded students, such as in financial management, preparing research/funding proposals, developing research ideas/questions, leading research projects, and supervising other students. Conversely, they reported being more involved in activities related to critical and creating thinking, strategic planning and advice, data collection, digital activities, research integrity/ethical conduct, and entrepreneurship. Collectively, these findings do not show systematic weaknesses in the training provided to students hired with IG/IDG grants, but instead point to variability in the types of experiences provided.

Career outcomes

SSHRC funding was often associated with career benefits for trainees. Based on the student survey, there were no significant differences in employment outcomes between IG/IDG-funded students and those who benefited from SSHRC scholarships and fellowships, but IG/IDG-funded students generally had more favourable employment outcomes than students who did not receive any SSHRC funding (i.e., unfunded students). Most notably, IG/IDG-funded students are more likely to work full-time (75%) than unfunded students (68%), when controlling for the time since graduation (because employment status is positively associated with years since graduation). On average, IG/IDG-funded PhD students also earned more ($79,000) than unfunded PhD students ($68,000) in their current position. Although not all comparisons showed different employment outcomes for SSHRC-funded students and unfunded students, receiving SSHRC funding was never associated with less favourable employment outcomes (e.g., lower salary or employment status). Similarly, focus group participants unanimously agreed that in no way had their IG/IDG experience been a hindrance to their careers.

Receiving SSHRC funding, whether through scholarship and fellowships or from IG/IDG, was not associated with a higher chance of working in particular sectors of employment. In other words, students were not more likely to work in the private, government, not-for-profit or academic sectors based on whether or not they received any SSHRC funding during their training. However, for those who were employed in the academic sector, those who received IG/IDG support at the master’s level or doctoral level were more likely to hold teaching faculty positions than unfunded students, and doctoral level trainees were more likely to hold research faculty, scientist, associate or fellow positions. Unfunded doctoral students were more likely than IG/IDG-funded doctoral students to hold adjunct faculty positions.

SSHRC-funded training opportunities were also consistently seen as being related to students’ continuing studies and/or their subsequent employment. About three quarters of IG/IDG-funded students believed that their current employment is highly related to their studies, significantly more than unfunded students (59%). And only 7% of IG/IDG-funded students reported there was little to no relation between their employment and studies, less than half the proportion of unfunded students (15%). Multiple examples were collected of IG/IDG-funded students who have gone on to pursue degrees on topics related to the IG/IDG projects they were involved in, and of students that have participated in further related researchin some cases as principal investigators or collaborators, thus contributing to enhanced research capacity. Indeed, some trainees have since been employed in positions that are directly related to their experience on IG/IDG projects, whether in academia or in other sectors (e.g., law, urban planning, education, museums and cultural organizations, community organizations, First Nations communities).

Moreover, the majority of IG/IDG-funded students (75%) indicated that their training was very useful in preparing for their career, and about one third pointed to their research/activities and personal/professional experiences as being factors that helped them obtain their current job. When explaining how their involvement in IG/IDG projects benefited their career trajectory, trainees primarily pointed to the skills, networks, and knowledge base they had acquired or enhanced, to SSHRC funding, and/or to their supervisor as the main factors that influenced their subsequent choices, and provided them with greater opportunities and a competitive advantage (see quotes from trainees, Table 5 ). Among survey respondents who indicated that they had taken a position “less than somewhat” related to the field in which they had studied, about 60% of IG/IDG-funded students pointed to the unavailability of jobs in their degree field as a factor in their decision to work in an unrelated field, significantly less than unfunded students (74%). Compared to IG/IDG-funded students, unfunded students were also more influenced into taking an unrelated job by factors such as pay and promotion opportunities, working conditions, job location, and change in career or professional interests. In short, the survey results are consistent with the perception that IG/IDG-funded training opportunities have opened doors to better career opportunities for trainees.

Table 5: In the words of trainees

“There is no doubt that my current job is the result of experience I gained with [my supervisor]. That knowledge and those skills are transferable to my work.”

“The skills and the knowledge base I gained that summer were very valuable when I started my own law practice, because I had an understanding of the historical development, the reasoning behind the law as it stands today. That quite literally gives you an enormous advantage in knowing how law operates the way it does.”

“Although I’ve really changed career paths, I carry the skill set with me, [to understand what’s happening in] different types of research, but also being able to facilitate focus groups. I didn’t anticipate that I’d keep doing that, but it’s come up quite a few times. I facilitate therapeutic groups and a lot of those skills started from that project.”

“The professors get to know the kind of person you are, and what kind of things are a good fit for you. Inevitably, when you get talking about career plans, they will often ask you if you’ve talked to ‘professor whatever’, and it’s useful. These people don’t just know what classes you took; they know you.”

“[My supervisor] put me in touch with someone who helped me with hierarchical modelling. This helped with how competitive I was to get into a clinical program.”

“People always notice when you have a SSHRC grant.”

Successful practices for training and employment outcomes

Among factors that contributed to successful training experiences and employment, participants in the student focus groups, interviews and case studies often highlighted opportunities to expand their professional network as one of the key benefits of participating in an IG/IDG project. Trainees provided examples of having formed valuable relationships not only with their own supervisor(s) on the project, but also with colleagues and national and international experts and other students that they met through the funded project, including at conferences. These professional networks had concrete impacts on employment outcomes, by establishing contact with professors or experts that helped them achieve their research goals. Networks were formed with people and organizations outside of academia as well (e.g., professionals, government, non-profit organizations, businesses), including through fieldwork experience. Indeed, more IG/IDG-funded students had the opportunity to participate in at least some inter-sectoral collaborative research (45%) than unfunded students (38%). Such connections were seen to have contributed to jobs opportunities and subsequent funding (e.g., grants, fellowships).

Positive working relationships within the research team, a variety of project-related tasks/activities, and a close alignment between the project and the students’ own interests and studies were other successful practices that contributed to research productivity (i.e., publications and other research outputs) and skills development, as observed in the focus groups and case studies. For example, in the context of larger studies, there may be several interesting facets for research that students can be given the opportunity to choose from, which they can pursue as their own research project in support of the larger study. Combined with SSHRC funding, these factors create an environment that helps attract highly qualified and motivated students at all stages of their academic careers. Overall, IG/IDG have often provided trainees with opportunities that align with the best practice principles for doctoral training identified by a group of European experts, including research excellence, attractive institutional environments, exposure to industry and other relevant employment sectors, international networking, and transferable skills training ^{Footnote 26}.

Importantly, the survey data presented above cannot be used to confirm a cause and effect relationship between SSHRC funding and employment outcomes, and there are limitations and potential biases associated with the survey data because respondents were identified through their supervisors. The fact that SSHRC does not have contact information for trainees hired with IG/IDG funding limits the ability to systematically follow-up with students funded through these grants. Other lines of evidence were used to corroborate survey findings in this evaluation, such as through examples from the focus groups. However, for future assessments, it would be useful to find approaches to collect more robust and representative information on training experiences and employment outcomes of IG/IDG funding. On a related note, not all students hired with IG/IDG funds were aware that the work they did for researchers was funded by SSHRC; given the value of declaring SSHRC-supported research experience to potential employers, some focus group participants suggested the importance of disclosing SSHRC funding could be better communicated to both researchers and students alike.

2.2.5. To what extent are the IG/IDG funding opportunities delivered in a cost-efficient manner?

As a first step to address this question, cost-efficiency analysis was used to assess the operational efficiency of the IG and IDG—in other words, the extent to which resources are being used as efficiently as possible. Other lines of evidence were also examined to explore other considerations and opportunities to improve the performance of the IG/IDG.

Cost-efficiency analysis

For the cost-efficiency exercise, financial data were used to determine the ratio of administrative costs to grant funding, comparing IG to its predecessor SRG and to the Natural Sciences and Engineering Research Council’s (NSERC) Discovery Grants (DG). IDG was compared to RDI only as no other program was similar enough to justify a comparison. The recent implementation of the IDG and IG funding opportunities made comparisons of their cost-efficiency ratios to those of other funding opportunities difficult because of the start-up costs associated with new funding opportunities and because of time lags between initial competitions and the distribution of funding. This limitation was mitigated by excluding the “transition years” between funding opportunities, although this means that comparisons are based on only a few years of expenditures for the new funding opportunities.

This analysis shows that IG and IDG are more cost-efficient than their respective predecessors, SRG and RDI. Administering IG costs $6.98 for each $100 of funds granted compared to $8.17 for SRG, and IDG costs $8.38 for each $100 of funds granted compared to $8.71 for RDI (Table 6). Although NSERC’s DG program was found to be more cost-efficient than IG ($4.38 for each $100 of funds granted), this is likely due to economies of scale available to a funding opportunity with a much larger total budget ($329 million for DG vs. $66 million for IG, on average). The same trends are observed if the comparisons are based on the percentage of administrative expenditures to total expenditures.

Table 6: Average cost-efficiency ratios for IG, IDG and comparable programs, 2005-06 to 2014-15

Expenditures	IG (2013-14 to 2014-15)	SRG (2005-06 to 2011-12)	DG (2007-08 to 2014-15)	IDG (2012-13 to 2014-15)	RDI (2005-06 to 2010-11)
Direct Salary	$1,558,031	$1,846,610	$3,411,463	$667,688	$50,648
Direct Non-Salary1	$309,094	$448,081	$1,157,524	$53,320	$17,044
Indirect and Direct Non-Attributable2	$2,265,662	$3,961,100	$8,611,235	$657,804	$114,702
Grant Expenditures	$61,600,539	$76,670,017	$315,756,544	$16,591,968	$2,136,773
Ratio of Administrative Expenditures to Grants ((Total Admin/Grants)x100)	$6.98	$8.17	$4.38	$8.38	$8.71
% of Admin Expenditures to Total Expenditures (Admin/(Admin+Grants))	6.51%	7.55%	4.01%	7.72%	8.01%

1. Consists of costs for transportation, telecommunications, publishing, professional and special services, printing, etc.
2. Consists of general administrative costs incurred by the SSHRC such as expenses generated by other divisions, like the Audit, the Corporate Affairs and Common Administrative Services Directorate (CASD).
Source: SSHRC financial data for IG, SRG, DG, IDG; NSERC financial data for DG

Looking at annual trends, the new funding opportunities also appear to have lowered their ratios of administrative expenditures to total costs for 2014-15, to levels well below those of the SRG and RDI funding opportunities (Figure 3). This analysis would need to be repeated with more years of data to confirm if this is a sustained trend or just a short-term annual variation.

Figure 3: Average Percentage of Administrative Expenditures to Total Costs for IG, IDG and Comparable Programs, 2005-09 to 2014-15

Description of figure 3

Figure 3. Average Percentage of Administrative Expenditures to Total Costs for IG, IDG and Comparable Programs, 2005-09 to 2014-15

This line graph presents the average percentage of administrative expenditures relative to total costs for Insight Grants, Insight Development Grants and comparable programs from 2005-06 to 20014-15. The y-axis shows the average percentage, starting at zero per cent and increasing in increments of one per cent, going up to ten per cent. The x-axis shows each year, starting at 2005-06 and going until 2014-15.  The programs represented are SRG, RDI, IG, IDG and DG. RDI begins as the highest, at nine per cent, and drops to seven per cent over two years, then remains around eight per cent until returning to seven per cent in 2011-12. SRG begins at seven per cent in 2005-06, spikes at just over eight per cent in 2008-09, then remains just under eight per cent for the rest of the graph. For both SRG and RDI, there is no data past 2011-12. IG begins at just under eight per cent in 2013-14 and drops to five per cent in 2014-15. IDG begins at nine per cent in 2012-13 and steadily drops to six per cent in 2014-15. DG begins at four per cent in 2008-09 and remains nearly constant to 2014-15.

The operation of these funding opportunities also involves non-monetary costs, most of which consists of volunteer time ^{Footnote 27}. The ratio of estimated non-monetary costs for 100$ of funded grants was slightly higher for IG ($5.83) than for SRG ($5.31). However, the ratio for IDG ($8.33) was much higher than that of the RDI program ($1.26). The higher ratio for IDG could be due to the need for a greater number of volunteers to review nearly five times as many IDG applications as were received for RDI, on average per year (922 vs. 185). Again, longer-term data would be needed to confirm these trends.

Opportunities to minimize costs while maintaining quality

When renewing its PA—which led to the creation of the IG and IDG—SSHRC was concerned with making processes more efficient and made several changes aimed at reducing the use of resources. Some of these improvements were also informed by the previous evaluation of the SRG/RDI (e.g., streamlining of the application process). When asked about potential areas for further efficiencies, all interviewed staff pointed to changes that have already been implemented with a mind towards reducing costs (monetary and non-monetary). Examples of these actions include the simplification of the program structure, the use of teleconferencing for the adjudication of IDG proposals instead of face-to-face meetings, eliminating the Notice of Intent to Apply stage of the IG applications, and introducing the use of the Canadian Common CV (in progress). Some case study participants provided favourable views of the efficiency of IG/IDG overall, including some specific features (e.g., online applications).

The majority of surveyed researchers also agreed that the amount and length of the IG and IDG funding opportunities were appropriate for their research, and that an appropriate amount of funding was earmarked for new or emerging scholars. Offering longer grants for the IG (up to 5 years) may also alleviate some of the pressure on applicants, as successful applicants will need to spend less time applying for funding. In short, SSHRC has implemented many changes to improve the efficiency of these funding opportunities, and no further suggestions were made to increase their cost-efficiency.

That said, some program stakeholders saw a need for further review of some of the recent changes to ensure they truly are maintaining or improving IG and IDG’s performance—the use of teleconferencing for IDG adjudication committee meetings being their main cause for concern. The majority of survey respondents with experience participating in adjudication committees for SSHRC perceived the quality of adjudication committee meetings to be “substantially or somewhat” negatively impacted by teleconferencing (60-81%), a view which was echoed by a few case study participants and key informants. Further evidence would be necessary to determine more precisely how and to what extent the quality of the adjudication process may have been affected. It would therefore be worthwhile, as part of future reviews of SSHRC’s peer review processes, to examine in more detail whether factors such as changes in format (e.g., teleconference vs. in person) has had any effects on the adjudication process quality and outcomes. On a related point, some key informants suggested that SSHRC could consider ways to improve the recruitment and/or training for merit committee members.

There also remains room for other improvements to the application process for these funding opportunities. Only about half of surveyed researchers reported a positive experience with the application process for the IG/IDG (e.g., ease and length of time to complete forms), and key informants also suggested ways should be found to reduce the burden on applicants.

2.2.6. What has been the impact of the delivery changes introduced with IG/IDG?

Both IG and IDG were developed as part of the major renewal of SSHRC’s PA. Through the PA renewal, SSHRC sought to reduce the complexity and overlap in its available funding opportunities, by providing fewer, more coherent and more flexible options to support SSH research. The implementation of these new funding opportunities—IDG in 2011-12 and IG in 2012-13—as successors of the SRG and RDI is associated with a range of design and delivery changes. Compared to their predecessors, IG and IDG have introduced refocused objectives, higher maximum grant values, longer maximum duration of grants (IG only, from three to five years), a new adjudication committee structure (including a new mechanism for multi-disciplinary adjudication), and a new scoring system for evaluating proposals (see Appendix 1 for details). Notably, to address historical issues with the relatively low number of SSHRC grants awarded to new scholars, the IDG also places less emphasis on researchers’ track records and reserves a proportion of funds (e.g., 50% in 2015) for emerging researchers.

Collectively, these changes have the potential to influence both the reach of these funding opportunities (e.g., success rates, demographics) and the achievement of expected outcomes. The following findings are based on a relatively short timeframe and should be interpreted with caution, particularly for any impacts of the delivery changes on intermediate and longer term outcomes.

Impact on reach

Among the most striking consequences of the implementation of IG/IDG to date is the five-fold increase in the number of applications received by SSHRC for IDG compared to RDI, from about 200 per year to over 1000 (Table 7). This was accompanied by a similar increase in the number of awards, from 50 per year to about 300, for a total of about $17 million in grants per year (compared to about $2 million for RDI). With the rising number of applications since its implementation, the IDG success rate has decreased slightly, but, on average (31%), remains higher than that of the last year of the RDI (30%). By all measures, the IDG has reached more SSH researchers than the RDI.

In contrast, the transition from SRG to IG has been associated with decline in the numbers of applications and grants (Table 8). Because of the higher maximum value of the IG ($500,000) compared to SRG ($250,000), project budgets in applicants have nearly doubled under IG, while the amount of available funds has not increased sufficiently to meet the demand. This situation has resulted in fewer—but larger and longer—IG projects being funded. The success rates have thus dropped from 37% in the last year of the SRG to about 24%, on average, for IG. In light of this situation, SSHRC has been working with its stakeholders to develop and implement a multi-pronged approach to increase the success rate for upcoming IG competitions. In particular, SSHRC has limited the maximum total grant value to $400,000 (instead of $500,000) as of the October 2015 competition and encourages review committees to trim application budgets. IG program managers at SSHRC are actively monitoring the impacts of these adjustments.

Table 7: RDI/IDG applications, awards and grant expenditures, 2009-10 to 2014-15

	Competition year	# of applications	# of awards	% of applications funded	Total value of grants awarded
RDI	2009-10	224	50	22.3%	$1,812,923
	2010-11	183	55	30.1%	$2,050,351
IDG	2011-12	630	246	39.0%	$13,096,445
	2012-13	934	327	35.0%	$16,519,335
	2013-14	1,024	304	29.7%	$18,268,799
	2014-15	1,128	285	25.3%	$17,408,342

Source: SSHRC’s Awards Management Information System (AMIS)

Table 8: SRG/IG applications, awards and grant expenditures, 2009-10 to 2014-15

	Competition year	# of applications	# of awards	% of applications funded	Total value of grants awarded
SRG*	2009-10	2,879	941	32.7%	$77,070,009
	2010-11	2,714	984	36.3%	$70,848,601
	2011-12	2,749	1,017	37.0%	$74,431,669
IG	2012-13	1,799	486	27.0%	$83,782,504
	2013-14	2,183	469	21.5%	$90,589,777
	2014-15	2,144	494	23.0%	$98,243,079

Note: *SSHRC issued 342 one year SRGs over competition years 2010 and 2011.
Source: SSHRC’s Awards Management Information System (AMIS)

Recall that the IDG was created in part to address the gaps in available funding for emerging researchers. And emerging researchers have responded to this opportunity: they have accounted for 65% of IDG applications since 2011-12 (i.e., both funded and unfunded). Meanwhile, while the proportion of applications from emerging scholars dropped from 37% for SRG to 16% for IG from 2009 to 2014 (no data on scholar type were available for RDI). This may also in part explain the declining number of IG applications, as emerging scholars shift their sights towards the IDG competitions.

This shift is not surprising, since emerging scholars are also more likely to be funded through the IDG than the IG (Figure 4 ). Since its inception, 66% of IDG recipients (i.e., funded applicants) have been emerging scholars, who benefit from the same success rate as established researchers for this funding opportunity (31%). In contrast, the vast majority of IG recipients have been established researchers (90%), while emerging researchers received the remaining 10%. Emerging scholars also have a lower success rate in IG competitions (15%) than established researchers (26%).

Figure 4: Principal Investigator Scholar Type, by Application Status for SRG, IG and IDG, 2009-2014

Description of figure 4

Figure 4. Principal Investigator Scholar Type, by Application Status for SRG, IG and IDG, 2009-2014

This bar graph represents the type of principal investigator (emerging or established) and the application status (unfunded or funded), by program, from 2009 to 2014. The programs depicted are SRG, IG and IDG.

Among unfunded SRG scholars, 62 per cent were established scholars and 38 per cent were emerging. Among funded SRG scholars, 66 per cent were established and 34 per cent were emerging. Among unfunded IG scholars, 82 per cent were established and 18 per cent were emerging. Among funded IG scholars, 90 per cent were established and 10 per cent were emerging. Among unfunded IDG scholars, 35 per cent were established and 65 per cent were emerging. Among funded IDG scholars, 35 per cent were established and 66 per cent were emerging.  

As noted above, success rates for IDG have generally been higher than for its predecessor (RDI) and this is most clearly the case for younger researchers: 38% of IDG applicants age 30 to 39 were granted an award, compared 21% of RDI applicants in the same age group. This coincides with an increase in the proportion of applicants from this age group (32% for RDI vs. 37% for IDG), such that this group represents about half of IDG recipients (49%), double the share of RDI recipients (25%). Moreover, a third of IDG recipients are between the ages of 40 and 49. Thus, despite the fact that fewer emerging researchers apply for and receive IG awards, overall these findings support the view of key informants, namely that the implementation of the IDG has effectively provided access to funding for younger researchers and emerging scholars.

The introduction of IG and IDG is associated with other favourable shifts in the demographics of applicants and/or recipients. First, compared to SRG, there are fewer researchers working alone on IG projects (69% vs. 59%), although this proportion increased for IDG compared to RDI (49% to 59%). Second, compared to RDI, the IDG funding opportunity receives a greater share of applications from principal investigators in small and medium institutions. The success rates of this group of applicants are also higher under IDG, thus shrinking the gap with success rates of applicants from large institutions. As such, one third of IDG projects are led by researchers from small and medium institutions, compared to about 12% of RDI projects. Since researchers from large institutions have traditionally led the vast majority of SRG awards (80% from 2009-2012) and continue to do so for IG (81%), IDG appears have increased access to SSH research grants for principal investigators in small and medium institutions.

Looking at other demographic variables, there have been limited impacts of the delivery change on the applications or awards across disciplines, regions, genders, or language of application.  One noteworthy impact, however, is related to funding for Aboriginal research. Prior to the program architecture renewal, the Aboriginal research committees (Relevance and Adjudication Committees) were run as part of an Aboriginal Research Pilot Program (with competitions offered in 2004, 2005, 2007 and 2009). With the transition to IG/IDG in 2011 and 2012, Aboriginal research was integrated into those two mainstream funding opportunities with the allocation of regular funds and yearly committee adjudication meetings. With the Aboriginal committees for IG/IDG running on a continual basis, SSHRC increased the representation of Aboriginal researchers within its adjudication process and stabilized funding for Aboriginal research. With the transition from the Aboriginal Research Pilot Program to the IG/IDG funding opportunities, the number of applications and awards for Aboriginal research increased significantly.  The Aboriginal Research Pilot (2004, 2005, 2007, and 2009) awarded 108 Grants totalling $17 million while IG/IDG (2011-2015) has awarded 142 Grants totalling $22 million.

The implementation of IG and IDG appears to have had very modest impact in terms of researcher satisfaction. The researcher survey pointed to some small differences, whereby IG/IDG recipients were more likely than SRG/RDI recipients to agree that adjudication criteria, funding amounts, and length of funding are appropriate. These differences are small, so it is safe to conclude that IG/IDG recipients are at least as satisfied with their funding opportunities as SRG/RDI recipients. Key informants and case study participants also generally had positive feedback on the changes to these funding opportunities (e.g., longer funding period for IG allows for more longitudinal studies), but also pointed the challenges raised by the lower success rates in the initial IG competitions.

Impact on outcomes

The evidence available to date does not show a clear impact of shifting from SRG/RDI to IG/IDG on the achievement of expected outcomes—either negative or positive. To summarize the findings presented below, differences between SRG/RDI and IG/IDG in terms of research outputs, knowledge mobilization, capacity building, and student training are generally small and inconsistent; they are also based on relatively premature and limited evidence. Because the grants being funded under IG/IDG are often larger and longer than under their predecessors, it is also challenging to determine whether any observed differences are due to grant size and/or project length, or to the funding opportunities themselves. Indeed, about 20% of IG/IDG awards totaled $200,000 or more (over three to five years), compared to less than 1% of SRG/RDI grants (over one to three years). The higher proportion of “high value” grants for IG increases the potential for unexpected grant size effects, as discussed below. Combined with the results presented in previous sections, these data show that the new funding opportunities appear to be achieving immediate outcomes to a similar extent as the SRG/RDI, and are on track to achieve longer-term outcomes. Again a longer timeframe is required to assess more precisely the impact of the delivery change on the achievement of outcomes.

Some differences in research outputs and knowledge mobilization strategies were observed between SRG/RDI and IG/IDG in the researcher survey, but could not be validated across other lines of evidence because it is too soon for most IG/IDG projects to have contributed to intended outcomes. Taken collectively, the observed differences are inconsistent. For example, IG/IDG recipients reported higher number of actual and planned knowledge mobilization strategies as well as higher rates of maintained or increased intensity for ongoing collaborations (i.e., after their projects are completed), but SRG/RDI recipients reported higher number of actual and planned research outputs, as well as higher expected use of research results by stakeholders. However, these effects were small, and grant size and/or duration play larger roles in the differences than the funding opportunity per se, as confirmed by regression analysis that tested for associations in the data. Most RDI and IDG respondents also believed SSHRC funding helped them develop new research questions, tools, methods, theoretical approaches, and collaborations, regardless of funding opportunity; in other words, there appear to be no differences between how RDI and IDG contribute to building research capacity.

Differences in training and employment outcomes for students who were hired through SRG/RDI or though IG/IDG were again small or inconsistent. It is also relatively premature to observe some of these outcomes for IG/IDG students (e.g., career benefits). With regard to their training experience, more IG/IDG-funded students reported being involved in personal-professional activities, but less in teaching activities, and were less satisfied with their teaching experience than SRG/RDI-funded students. There were no differences in their participation in various types of research activities, but IG/IDG-funded students reported having produced slightly fewer oral and poster presentations at national and/or international conferences than SRG/RDI-funded students. There were little or no differences in employment outcomes (e.g., slight difference in employment sector, but none for employment status or income), and time since graduation clearly overshadowed any effect of the funding opportunity on these outcomes. However, IG/IDG-funded students were less likely to find that training received had helped them prepare for their career (53% vs. 78% for SRG/RDI-funded).

Unexpectedly, the funding allocated to trainees appears to have decreased in the transition from SRG/RDI to IG/IDG, based on the total proportion of grant funds allocated to salaries and stipends (as per statements of account submitted to SSHRC), and on the average funding reported by surveyed students. Further long-term evidence would be needed to assess this, but the survey data suggest this decrease may be associated with a shift in the types of students funded via the grants: whereas SRG/RDI funding was used to hire a large share of PhD students, IG/IDG funding may be used to hire more equal shares of students across all academic levels (e.g., undergraduate, masters and doctoral). This potential decrease does not seem to have affected student satisfaction: the proportion of funded students who were satisfied with the amount of money received was similar across those hired through former and current funding opportunities (75% vs. 80%), and IG/IDG-funded students reported more satisfaction with length of training than those hired under SRG/RDI. Moreover, final research reports suggest that more HQP in general, and more types of students and non-students, tend to be hired for projects with larger grant amounts.

To better understand the impact of larger grant sizes, data from the researcher survey, final research reports, and literature were revisited. The researcher survey and final research reports confirm that larger grants are associated with more research outputs, more knowledge mobilization strategies used, more partnerships and research infrastructure development, and more hiring of students. However, most of these associations could only be assessed for grants up to $250,000 (the maximum value of SRG), and some of them appeared to show a “plateau effect” at which outputs or outcomes ceased to increase in proportion to the grant amount. For example, doubling the grant size did not automatically double the number of HQP hired, and the average number of research outputs produced with $100,000-$150,000 grants (30) was actually higher than the average produced with larger, $150,000-$200,000, grants (28). SRG data also indicate that larger average grant sizes and expenditures are associated with larger amounts and proportions of surplus grant funds (i.e., unspent funds). All of these analyses would need to be repeated once more data on the larger IG/IDG grants are available to assess potential grant size effects for awards of more than $250,000.

While such quantitative comparisons do not provide much insight on differences in the quality of outputs or on their contribution to outcomes, they do suggest that bigger is not necessarily better in the context of research funding. This conclusion was echoed by some key informants, especially those who considered that larger project budgets, in the context of limited available funds and higher numbers of applications, results in insufficient resources to fund every worthy project.

Similarly, several articles examined in the literature review suggested that the size of research grants may increase to a point at which there are diminishing returns, such as in terms of outputs, citations or impact. ^{Footnote 28,Footnote 29} The literature provided a range of examples of possible effects of increase grant sizes, both positive (e.g., more visibility for larger projects ^{Footnote 30}) and negative (concentration of funding across a small number of researchers ^{Footnote 31}, a lack of diversity in funded areas/topics, evaluators may be more conservative when dealing with larger grants ^{Footnote 32}). Such effects were often found to vary not only across disciplines (SSH, NSE, health), but also within different fields of SSH research, such that there is no optimal grant size that can likely be recommended. Indeed, the general consensus across these studies is that optimal funding approach should involve a mix of different instruments, and not exclusively large-scale grants.

---

3.0 Conclusions

Relevance

There is an ongoing need for SSH research to answer questions on issues that matter most to Canadians, such as those relating to culture, business, education, health, the environment and policy. SSHRC’s IG and IDG are two of the main funding opportunities currently available to SSH researchers in Canada, and the few alternatives that were identified had more limited accessibility and scope. Without IG/IDG funding, proposed SSH research was often not completed as planned. In this context, there continues to be a high level of demand for these grants, and IDG funding in particular fills a unique need for emerging researchers across Canada.

IG/IDG objectives are aligned with federal government priorities (e.g., Seizing Canada’s Moment: Moving Forward in Science, Technology and Innovation 2014) and directly support SSHRC’s primary strategic outcome for 2013-16, which is “to make Canada a world leader in SSH research and research training.” Providing IG/IDG funding is also aligned with the roles and responsibilities of the federal government when considered alongside the roles and responsibilities of provincial/territorial governments and of industry. The federal government has been funding SSH research for over 50 years, and it is appropriate that it continues to do so in light of the critical importance of research to national economic development, as the world becomes increasingly knowledge based.

Performance

The evaluation concludes that the IG/IDG and their predecessors (SRG/RDI) are contributing to the achievement of their intended immediate and intermediate outcomes, namely the dissemination and potential use research results, strengthened research capacity, and training of HQP that is associated with career benefits. In the longer term, all of these are likely contributing to the recognition of the expertise and excellence of Canada SSH research.

With regard to knowledge mobilization, there is active dissemination of, and broad interest in, the research supported by the IG/IDG funding opportunities (and predecessors), primarily within, but also outside of academia. Compared to unfunded projects, funded projects generally produced more outputs, employed more knowledge dissemination strategies, and were more likely to have their results used. However, there is limited information available on stakeholder use, and even less so on the subsequent benefits of project results. Although SSHRC funding was a key enabler of knowledge mobilization, further efforts to support these activities and recognize their value could help mitigate barriers to the uptake and use of research results that persist within the academic system.

SSHRC funding, particularly through the RDI/IDG, has also strengthened research capacity in the majority of funded projects by supporting and enhancing collaborations (established and new), and by enabling the development of new research questions. Those who received RDI/IDG funding are more likely to receive future SSHRC funding, and higher amounts of subsequent funding from SSHRC and other sources.

The evaluation finds that IG/IDG funding also supports training for large numbers of students and other HQP. Funded projects often provide opportunities for networking and the development of a variety of research and professional skills applicable to careers in a variety of areas. These training experiences are associated with positive employment outcomes for students, based on comparisons with employment outcomes of students who did not receive any SSHRC funding. These findings are corroborated by multiple examples, but could be strengthened through the systematic collection of representative information on training experiences and subsequent employment outcomes.

As for longer-term outcomes, SSHRC funding was linked to various measures of research excellence, such as Canada’s high scientific impact and high standing in SSH research internationally. Researchers funded through the IG/IDG (and predecessors) have received more prestigious prizes than unfunded researchers.

The evaluation also finds that IG and IDG are both delivered cost-efficiently, particularly in comparison to their respective predecessors, SRG and RDI. SSHRC has implemented many changes to improve efficiency since the last evaluation, but there remains room for other improvements to the program (e.g., application and review process).

Finally, the delivery changes have had mixed results in terms of the reach of these funding opportunities: while IDG have effectively provided access to funding for younger and emerging researchers, success rates have dropped for IG compared to its predecessor SRG as IG project budgets have outpaced available funding. To date, observed impacts of the delivery change on outcomes are small and inconsistent, and a longer timeframe is required to more accurately assess these impacts and recent adjustments made by SSHRC. Overall, IG/IDG appear to be achieving immediate outcomes to a similar extent as their predecessors SRG/RDI, and are on track to achieving long-term outcomes.

---

4.0 Recommendations

Based on the findings and conclusions of the current evaluation, four recommendations are proposed for consideration regarding SSHRC’s IG and IDG.

The evaluation demonstrated the continued need for SSH research and for the IG/IDG funding opportunities specifically to support SSH researchers in Canada. Few equivalent alternatives exist, and IDG funding in particular fills a unique need for emerging researchers. IG/IDG objectives were also found to align with current federal priorities, roles and responsibilities. In terms of their performance, these funding opportunities are contributing to the achievement of expected immediate and intermediate outcomes, and more delivered more cost-effectively than their predecessors.

Limitations in the available evidence on the use of IG/IDG-funded research results by stakeholders— particularly on use that may occur after the end of the grant—suggests a need for a better approach to monitor the uptake and use of grantees’ research. This approach should allow for systematic follow-up on funded projects after at specific time period, and consider ways to collect information or evidence that complements the perceptions of grantees (e.g., citations in peer-reviewed and grey literature, input from knowledge users, web or social media data).

Given the barriers to the broad uptake and use of research results that persist within the academic system, SSHRC should also continue to encourage and promote the value of researchers undertaking knowledge mobilization activities that are intended for stakeholders outside of academia. Such measures should include sharing best practices and recognizing knowledge mobilization activities as one measure of research and/or researcher excellence. However, such efforts should not be limited to SSHRC: researchers and their institutions should also be encouraged to promote, support, and enable the sharing of research results both within and outside academia.

Activities to support knowledge mobilization outside of academia could also benefit from greater and/or specific funding. SSHRC could consider additional funding linked to the IG/IDG projects for knowledge mobilization outside of academia. Efforts to monitor the uptake and use of research results should also cover these additional activities.

The evaluation encountered issues with collecting and interpreting information on training experiences and career outcomes because SSHRC does not have contact information for trainees hired with IG/IDG funding. This information would help not only future performance monitoring efforts, but also inform future efforts by program management to improve the consistency of training across all grants, for example of identifying and communicating examples of best practices of grantees regarding the training of HPQ (including for students, postdoctoral fellows and other research trainees). However, as an important step to examine the feasibility of collecting contact information from students funded through grants, SSHRC would be required to conduct a privacy impact assessment.

A longer timeframe is required to determine the impact of the IDG and IG funding opportunities on the achievement of expected outcomes because of the recent implementation of the IG/IDG, the duration of IG (up to five years), and the fact that many outcomes can only be measured after the end of grants. To date, IG/IDG appear to be achieving immediate outcomes to a similar extent as their predecessors SRG/RDI, and are on track to achieving long-term outcomes. Indeed, the evidence available to date does not show a clear impact of shifting from SRG/RDI to IG/IDG on the achievement of expected outcomes—either negative or positive. Observed effects, often based only one line of evidence and short-term data, were generally small and inconsistent. These observations nonetheless justify further monitoring and/or validation of potential effects of the delivery changes over the longer term.

In addition, the evaluation has identified some actual and potential challenges associated with design and delivery of the new funding opportunities. For example, some stakeholders perceived that the use of teleconferencing for IDG adjudication committee meetings was affecting the quality of the adjudication process, although evidence of how or to what extent quality was affected was lacking. It would therefore be worthwhile, as part of future reviews of SSHRC’s peer review processes, to examine in more detail whether factors such as changes in format (e.g., teleconference vs. in person) has had any effects on the adjudication process quality and outcomes.

One of the key changes under IG/IDG is that grants are often larger and longer than under SRG/RDI. This makes it challenging to determine whether any observed differences in outputs and outcomes are due to grant size and/or project length, or to the funding opportunities themselves. The higher proportion of “high value” grants for IG also increases the potential for unexpected grant size effects, whereby larger grants appear to show a “plateau effect” or “diminishing returns”—a phenomenon documented in the literature—whereby outputs or outcomes (e.g., publications, student training) cease to increase in proportion to the grant amount. However, at this time, most of these associations could only be assessed for grants up to $250,000 (the maximum value of SRG), so all of these analyses would need to be repeated once more output and outcome data are available on the larger IG/IDG grants.

Another challenge faced by the IG/IDG funding opportunities, in the context of limited available funds, is that larger grant amounts results in insufficient resources to fund every worthy project. In the initial years of the IG implementation, fewer—but larger and longer—IG projects were funded. The success rates thus dropped from 37% in the last year of the SRG to about 24%, on average, for IG. SSHRC had identified this issue prior to the evaluation, and has begun working with stakeholders to develop and implement a multi-pronged approach to increase the success rate for upcoming IG competitions. In particular, SSHRC has limited the maximum total grant value to $400,000 (instead of $500,000) as of the October 2015 competition and encourages review committees to trim application budgets. As such, SSHRC should continue to proactively monitor the impacts of the IG/IDG design and delivery, as well as the impacts of recent changes and success rates, to inform program management.

---

Appendix A - Key Characteristics of IG/IDG and Predecessors

Table 9 compares key characteristics of the Insight Grant (IG) funding opportunity to its predecessor, the Standard Research Grants (SRG).Similarly, Table 10 compares key characteristics of the Insight Development Grant (IDG) funding opportunity to its predecessor, the Research Development Initiatives (RDI).

Table 9: Comparison of SRG and IG funding opportunities

Characteristics	Standard Research Grants (1960s – 2011/12)	Insight Grants (2012/13-present)	Summary of key changes for IG
Objective	To support research and develop excellence in research activities	To support research excellence	Refocused objective
Length	1-3 years	3-5 years	Longer duration
Value	≤ $250,000	$7,000 - $500,000 Footnote 1	Higher maximum value
Schedule	Annual	Annual
Notice of Intent	Optional	Mandatory Footnote 2
Type of scholar	Distinct categories for regular scholar or new scholar	Not specified in application	Career stage not considered
Committee structure	Submit directly to pre-established committees	Submit based on research group and committees are developed in response	New adjudication committee structure
Treatment of multidisciplinary applications	Single multidisciplinary committee	Many committees are multidisciplinary. An additional file reader is assigned if multidisciplinary evaluation is required	New mechanism for multi-disciplinary adjudication
Evaluation criteria	Evaluation based on record of achievement (60%) and program of research (40%) Footnote 3	Evaluation based on aim/importance (40%), feasibility (20%), and capability/expertise (40%)	New scoring system for evaluating proposals

Table 10: Comparison of RDI and IDG Funding Opportunities

Characteristics	Research Development Initiatives (1998 – 2010/11)	Insight Development Grants (2011/12-present)	Summary of key changes for IDG
Objective	To support research in its initial stages by supporting the development of new ways of analyzing, structuring, integrating and transferring knowledge	To support research in its initial stages, i.e., enable development of new research questions and experimentation with new methods, theoretical approaches, and/or ideas	Refocused objective
Length	1-2 years	1-2 years
Value	≤ $40,000	$7,000-$75,000	Higher maximum value
Schedule	Bi-annual	Annual	Single annual competition
Notice of intent	None	None
Type of scholar	Not specified in application	Distinct categories for emerging scholar or established scholar	New career stage consideration
Committee structure	Multidisciplinary committee	Submit based on research group and committees are developed in response	New adjudication committee structure
Treatment of multidisciplinary applications	N/A (single multidisciplinary committee)	All committees are multidisciplinary and an additional file reader is assigned if multidisciplinary evaluation is required	New mechanism for multi-disciplinary adjudication
Evaluation criteria	Evaluation criteria include: significance/originality; quality of plans; applicant’s qualifications; feasibility to meet RDI objectives; appropriateness of budget	Evaluation based on aim/importance (50%), feasibility (20%), and capability/expertise (30%)	New scoring system for evaluating proposals

---

Appendix B: Insight Program Logic Model

Description of Appendix B

Appendix B presents the Insight Program Logic Model. There are six text boxes stacked vertically. These represent the various outcomes.

The first two text boxes relate to objectives to which all SSHRC funding opportunities contribute.

The first text box is for SSHRC’s Strategic Outcome, which is that Canada is a world leader in SSH research and research training.

The second text box is for Long-term outcomes, Indirect influence. In this category are three points, indicated in their own boxes. The first is The expertise and excellence of Canadian SSH researchers is recognized on the national and international level. Second is Individuals and organizations in the public, private, not-for-profit and academic sectors benefit from SSH research knowledge. Third is Individuals with strong research, leadership and management skills are employed in positions in academia, government, private and not-for-profit sectors.

The next four boxes are specific to Insight funding opportunities.

The first text box is for Intermediate Outcomes, Direct and indirect influence. This includes three points indicated in their own boxes. The first is Research knowledge is produced and mobilized. The second is Research capacity is developed or strengthened within postsecondary institutions. The third is Trainees enhance their skills.

The second text box is for Immediate Outcomes, Direct influence. This includes three points indicated in their own boxes. First is SSHRC’s objectives and priorities are reflected in the response to its programming. Second is Individuals and teams engage in research and research-related activities. The third is Students and postdoctoral fellows are engaged in research training.The third text box is for Outputs, Direct control. This includes two points indicated in their own boxes. The first is Program literature, communication products, presentations, and other interactions with SSH community. The second is Grants for individuals and teams (SIG, ASU, IDG, IG).The last text box is Activities, Direct control. This includes three points indicated in their own boxes. The first is Program Development: Develop and regularly review funding opportunities and merit-review practices. Second is Program Delivery and Monitoring: Deliver funding opportunities, joint initiatives and co-funding agreements with partners; co-ordinate merit-review process; monitor and assess performance and results.

Third is Outreach and Engagement: Promote funding opportunities and SSHRC’s objectives and vision.

Blue upward arrows connect each of the text boxes to the one above.

Download Image

Insight Logic Model Description

The logic model presented above shows the relationship between key activities, outputs and outcomes for the Insight program. As funding opportunities under this program, IG and IDG contribute to the achievement of these outcomes, along with other Insight program funding opportunities. This section describes the Insight program logic model as it relates to the IG and IDG funding opportunities.

Activities

Activities are actions that are undertaken to produce program outputs. For the Insight program logic model, activities refer to the following ongoing work undertaken at SSHRC:

• Program development: Develop and regularly review funding opportunities and merit-review processes. New funding opportunities and the accompanying merit-review process are developed, as required. Existing funding opportunity descriptions, instructions, and merit-review practices are reviewed on a regular basis and updated as required, taking into consideration feedback from the community, observer’s reports, merit-review committee policy discussions, federal budget announcements, and lessons learned.
• Program delivery and monitoring: Deliver funding opportunities, coordinate merit-review process, monitor and assess performance and results. Funding opportunities are delivered in a timely manner within budgets, including coordinating merit review committees and results announcements. Once the grants are awarded, guidance is provided to institutions and awardees on post-award matters, such as the eligibility of expenses. Furthermore, the performance and results of the individual grants and of the funding opportunities are monitored and assessed.
• Outreach and engagement: Promote funding opportunities and SSHRC’s objectives and priorities. Information on SSHRC funding opportunities as well as the organization’s objectives and priorities is actively communicated to researchers, institutions, and other stakeholders. This may be done through a variety of ways, such as representing SSHRC at various events, organizing webinars, site visits or through phone conversations with potential applicants.

Outputs

The direct products or services generated from the Insight program activities include the following:

• Grants for individual researchers and teams. The outputs of the Insight funding opportunities developed, delivered, and promoted by SSHRC, are grants to individual and team researchers at Canadian postsecondary institutions in the social sciences and humanities to conduct research and research-related activities. In the case of IDG and IG, the grants are directly awarded to the individuals and teams.
• Program literature, communication products, presentations and other interactions with SSH community. A number of outputs other than grants are produced in order to achieve the desired outcomes of the Insight funding opportunities. Information about new and/or existing funding opportunities is communicated through literature aimed at applicants and merit reviewers to share the description, objectives, eligibility, application and review processes and evaluation criteria. Guidelines and policies are created or updated as required and integrated into the merit-review process to communicate SSHRC’s priorities and visions. For instance, guidelines for effective research training were developed to emphasize the importance that SSHRC places on student training and skill development.. Finally, through outreach and engagement, communication products are generated and advice is provided to potential applicants.

Immediate outcomes

These are the external consequences directly attributable to the outputs. They are expected to occur in the early stages of the award and may be measured in the applications and in end-of-grant reports.

• SSHRC’s objectives and priorities are reflected in responses to its programming. As a result of the program literature, communication products, presentations and other interactions, SSHRC stakeholders demonstrate an understanding of SSHRC’s vision and priorities. For instance, SSHRC has championed effective knowledge mobilization. As a result, it is expected that applicants will submit proposals with well-defined knowledge mobilization plans and merit reviewers will have a clear understanding of what an acceptable knowledge mobilization plan is.
• Individuals and teams engage in research and research-related activities. As a result of the funding provided through the IDG and IG funding opportunities, individuals and/ or teams of researchers engage in research and research-related activities. This may include developing and enhancing theoretical frameworks, methodologies and networks, as well as collecting and analyzing data.
• SSH students and postdoctoral fellows are engaged in SSH research training. As a result of the funding provided through research and research-related grants to individuals and teams, students and postdoctoral fellows are engaged in research and research-related activities. As per guidelines to applicants and merit reviewers, a research training component is included in almost all IDG and IG funded projects. In general, the salary of students and postdoctoral fellows are paid directly through the award, yet in some instances, students and postdoctoral fellows who are actively involved in a research project, may have their salary paid through a fellowship or another grant.

Intermediate outcomes

These are the external consequences that are expected to occur once the immediate outcomes have been achieved. They are expected to occur around the end of the award and may be measured in the end-of-grant reports.

• Research is produced and mobilized. As a result of the research engaged in by individuals and teams, new knowledge is produced and mobilized through a variety of means (e.g., publications, presentations, and social media) to academic and non-academic audiences. The knowledge produced has the potential to lead to intellectual, cultural, social, and economic benefit at this stage, because it is made available for knowledge receptors, both academic and non-academic. Given the different objectives of the funding opportunities, the expectations for the production and mobilization of knowledge are different. Since the IG funding opportunity funds mature research projects and IDG funds the initial stages of research, it is expected that there will be greater evidence of research contributions than through IG than IDG.
• Trainees increase their research, professional, and knowledge mobilization skills. As a result of their involvement in research projects, it is expected that students and postdoctoral fellows involved in IG and IDG projects will acquire professional skills specific to their field of study and skills to effectively conduct research, research training, and knowledge mobilization.
• Research capacity is developed or strengthened within postsecondary institutions. As a result of the research and research-related activities undertaken with IDG funding and the peer-review process that recognizes the importance of supporting emerging scholars, early-career researchers are expected to become more competitive in other grant competitions. While funding from the IG funding opportunity develops research capacity within Canadian postsecondary institutions, it is not an explicit objective.

Long-term outcomes

These are the external consequences that are expected to occur once the intermediate outcomes have been achieved. They are expected to occur after the end of the award. In some instances, they may be measured in the end-of-grant reports, but for the most part, they are best measured through evaluation, special studies, or other sources of data. The achievement of the long-term outcomes is dependent on the success of all of SSHRC’s funding opportunities, and for this reason, this set of outcomes is common across all of SSHRC’s performance measurement strategies. The achievement of the long-term outcomes is also dependent on favourable external factors.

• The excellence and expertise of Canadian SSH researchers is recognized on the national and international level. SSHRC aims to develop excellence and expertise in the social sciences and humanities in order to be able to anticipate and address current, emerging, and future issues of importance in Canada and internationally. As a result of the production and mobilization of research, and the development of research capacity within Canadian postsecondary institutions, individuals and teams funded through the IDG, and IG funding opportunities, have developed expertise on current and emerging issues of importance to society, and this expertise has been recognized through prizes and recognitions. While IDG and IG both contribute to the development of research excellence across all SSH fields, their contributions are proportional to the size of their budgets. While SSHRC funds a wide range of research through the IDG and IG (as well as other funding opportunities), neither SSHRC nor Canadian SSH researchers can predict with certainty the questions of future importance. Through its identification and promotion of Future Challenge Areas, SSHRC has attempted to mitigate this risk.
• Research knowledge creates new value in all sectors. SSHRC seeks to enable the creation and movement of knowledge involving interdisciplinary, inter-institutional, cross-sectoral and international perspectives to and from various sectors of society, in order to have an intellectual, cultural, social, and economic impact. As a result of individuals and teams producing and mobilizing their research, the research results arising from IDG and IG research and research-related activities are being used by stakeholders across the public, private, not-for-profit, and academic sectors. However, the contribution of each funding opportunity is proportional to the size of their budgets. It must be recognized that research is inherently risky and that for every paradigm-shifting insight, there are research projects that do not produce the expected results. This outcome is also dependent on potential end users of the research in non-academic sectors participating in and being receptive to the research process, as well as investing time and money in SSH research and its mobilization.
• Individuals with strong research, leadership and management skills are employed in positions in all sectors. SSHRC values student training and skill development because it produces highly qualified individuals who take on leading roles in all sectors of society. A supply of highly qualified personnel is essential for Canada to compete internationally in the knowledge economy. As a result of the broad set of skills acquired by trainees involved in IG and IDG, they are employed in positions that utilize their research and leadership skills in academia, government, private, and not-for-profit sectors. While SSHRC provides grants that involve training through IDG and IG (as well as other funding opportunities), and establish guidelines for effective training, skill development is dependent on how it is implemented by institutions, researchers, and trainees. The employment of trainees is also dependent on the job market at the time of graduation and the decision of trainees to enter the job market instead of pursuing additional training.

Strategic (or ultimate) Outcome

This is the longer-term benefit that may be reasonably and causally attributed to a certain extent to SSHRC’s programs, as a consequence of the intermediate and long-term outcomes having been achieved.

Ultimately, SSHRC’s funding through all of its programs and vision leads to Canada being a world leader in social sciences and humanities research and research training.

---

Appendix C: Evaluation Matrix

Question	Indicator	Admin data	Doc/lit review	KI interviews	Researcher survey	Case studies	Student survey	Student focus group
1. Is there a continued need for IG/IDG in light of the current context?	1.1) Description of demand for IG/IDG funding from 2009-2014 (trends in # and type of applicants, potential applicants and grant recipients)	✓	✓
	1.2) Description of similar funding sources available for SSH research		✓	✓
	1.3) Extent of alignment between IG/IDG objectives and perceived funding needs of SSH researchers		✓	✓
	1.4) Extent of alignment between IG/IDG objectives and perceived knowledge production/research needs of Canadians		✓	✓
2. To what extent are IG/IDG objectives consistent with federal government priorities and SSHRC strategic outcomes?	2.1) Extent of alignment between IG/IDG objectives and federal government priorities		✓	✓
	2.2) Extent of alignment between IG/IDG objectives and SSHRC strategic outcomes		✓	✓
3. To what extent is providing IG/IDG funding an appropriate role for the federal government?	3.1) Description of federal responsibility for providing SSH research funding (including division between federal and provincial/territorial responsibilities)		✓	✓
	3.2) Description of federal government’s historical involvement in similar grant programs		✓
4. To what extent have the IG/IDG funding opportunities contributed to the dissemination and use of research results?	4.1) Indicators of production of research results:  # and description of research outputs produced by type (total and average per grant)	✓			✓
	4.2) Indicators of dissemination of research results:  # and description of knowledge mobilization strategies used (total and average per grant)  % of projects receiving media attention	✓			✓
	4.3) Indicators of use of research results:  % of grantees reporting that their research results have been used by type of use  % of grantees reporting that they have been in contact with other stakeholders (e.g., academics, government, etc.) regarding use of their research  % of grantees who have developed partnerships (information or formal) related to the IG/IDG funded research  Examples of ways in which knowledge/products produced by IG/IDG funded research have benefited the public, private and non-profit sectors	✓		✓	✓	✓
	Description of knowledge mobilization success/hindrance factors			✓		✓
5. To what extent have the IG/IDG funding opportunities contributed to trainees developing skills that lead to employment?	5.1) Indicators of involvement in training: # of trainees involved in IG/IDG funded research by type  % of IG/IDG funds spent on trainees by type  Description of training/experience provided	✓		✓				✓
	5.2) Indicators of research/professional skill development  # and type of trainee research outputs (e.g., publications, conference presentations, etc.)  % of trainees in R.A., T.A., or teaching positions  Perceptions of the extent of skill development  Perceptions of quality of training experience			✓			✓	✓
	5.2) Indicators of employment  % of previous trainees employed in academic sector  % of previous trainees employed in other relevant fields (non-academic) % of previous trainees reporting that their degree/fellowship is related to their current employment % of previous trainees who believe their training was helpful in preparing for their career						✓	✓
	Description of student employment success/hindrance factors			✓		✓		✓
6. To what extent have the IG/IDG funding opportunities contributed to the expertise and excellence of Canadian SSH researchers being recognized nationally and internationally?	6.1) % of funded projects cited for Canadian and/or international recognition or prizes	✓			✓
	6.2) Correlation between IG/IDG funding and university rankings		✓
	6.3) Canada’s ARC and ARIF in comparison to other countries with a similar number of English language SSH publications		✓
	6.4) Alignment between IG funded research and future challenge areas	✓	✓
	6.5) Description of success/hindrance factors to research excellence			✓		✓
7. To what extent has the IDG funding opportunity contributed to strengthened research capacity?	7.1) # and description of new research questions, methods, or theoretical approaches developed	✓		✓	✓
	7.2) % of grants that led to new collaborations				✓
	7.3) % of respondents who believe capacity development would not have been possible without IDG funding				✓
	7.4) % of IDG recipients to be awarded other funding following IDG completion				✓
	7.5) Description of research capacity development success/hindrance factors			✓		✓
8. To what extent are the IG/IDG funding opportunities delivered in a cost-efficient manner?	8.1) Ratio of administrative costs to grant funding for IG/IDG and comparable programs	✓	✓
	8.2) Ratio of non-monetary costs (e.g., volunteer and applicant time) to grant funding for IG/IDG and comparable programs	✓	✓
	8.3) Opportunities to minimize cost to stakeholders (money and time) while maintaining quality			✓
9. What has been the impact of the delivery changes introduced with IG/IDG?	9.1) Description of changes to funding opportunity delivery		✓	✓
	9.2) Comparison of SRG/RDI reach to IG/IDG reach (i.e., success rates, recipient demographics, number and involvement of collaborators, research areas)	✓	✓
	9.3) Comparison of SRG/RDI short term outcomes to IG/IDG short term outcomes (i.e., research outputs, trainees involved, new research questions, methods, and theoretical approaches)	✓			✓		✓

---

Footnotes