Evaluation of the Storytellers Challenge

Produced by R.A. Malatest & Associates Ltd.

The Honourable François-Philippe Champagne, P.C., M.P.
Minister of Innovation, Science and Industry

Cat. No. CR22-125/2023E-PDF
ISBN 978-0-660-48735-9

Evaluation of the Storytellers Challenge
May 2023

(PDF, 682 KB)

Executive summary
1. Introduction
2. Findings
3. Recommendations
Appendix A—Methodology

Executive summary

The annual Storytellers Challenge, launched by the Social Sciences and Humanities Research Council (SSHRC) in 2013, asks postsecondary students to develop short presentations that show Canadians how SSHRC-funded research is impacting our lives, our world and our future for the better. Entrants can submit a three minute audio or video presentation, or a 300 word text or infographic document featuring a research project or initiative funded by SSHRC. The Top 25 submissions receive $3,000 and the opportunity to present their submissions at the Congress of the Humanities and Social Sciences annual national conference.

As it approached its 10-year anniversary, SSHRC initiated an evaluation of the Challenge to assess: the extent to which it was seen as addressing a key need; the level of awareness of Challenge among postsecondary students; the inclusivity of the Challenge; and participant satisfaction with the Challenge. The evaluation was guided by the following nine evaluation questions:

Is there a continued need for the Challenge, with regard to its main goal (i.e., the promotion of the importance of SSHRC-funded research and its impact)?
Do the broader societal trends in terms of the consumption of information over the past decade (evolution of media and formats) affect the relevance of the Challenge? Has the Challenge capitalized on the growth and expansion in media (technology, capabilities, usage)?
To what extent are the targeted institutions and participants aware of and participating in the Challenge?
What do key stakeholders (entrants, finalists, postsecondary institutions, judges, Challenge and SSHRC management) identify as strengths and weaknesses of the Challenge?
Is the Challenge designed and delivered in a manner so as to encourage equity, diversity, and inclusion?
To what extent has the Challenge contributed to increasing awareness of the value and impact of SSHRC-funded research across Canada?
To what extent is the Challenge contributing to the communication of research results in the social sciences and humanities?
To what extent have participants gained valuable experience and enhanced their skills with regard to communicating the relevance and impacts of social science and humanities research?
To what extent is the design and delivery of the Challenge efficient?

Multiple lines of evidence were collected to answer the evaluation questions, including: a review and analysis of administrative data and documents; a review of other similar competitions; key informant interviews with entrants, judges, workshop instructors, participating institutions, SSHRC staff, and Congress staff; and surveys with entrants, judges, workshop instructors; and participating institutions. Different lines of evidence enabled the evaluation to triangulate its findings and provide a more comprehensive review.

Findings

Continuing Need for Storytellers

The Storytellers Challenge fits with the mandate of SSHRC to promote and support postsecondary-based research and training in the social sciences and humanities. It also aligns with the SSHRC’s Strategic Priorities of: contributing to innovative solutions to global challenges and opportunities by sharing research; and demonstrating its value and growing Canada’s research talent by developing skills in communication and knowledge mobilization. Further it was widely reported that Storytellers was fulfilling an important need within the research community. Storytellers allowed for the sharing of research findings with the broader research community and, to a lesser extent, the general public. Storytellers also promoted the development of research communication skills, viewed as increasingly important for the careers of new researchers.

While it was acknowledged that other research communication competitions existed, including Three-minute thesis, they were more restrictive in their entry requirements than Storytellers. Storytellers has both broader eligibility criteria for potential entrants, and allowed for a wider range of submission formats. This allowed for more Canadian postsecondary students to enter the Challenge and gave them greater freedom in determining how best to present their findings. Some of the entrants further noted that the flexibility of submission formats encouraged creativity among applicants.

Developing Participant Skills

While Storytellers was generally perceived as helping in the development of research communication skills, entrants reported uneven benefits. Most entrants (55%) indicated that their involvement in Storytellers improved their research communication skills. Entrants who advanced to the Top 25 were invited to participate in a research communication workshop. As such, the Top 25 reported more benefits from their participation, with 63% reporting improved research communication skills, 57% indicating they were better able to convey complex ideas, and 61% saying that attending the workshop was a valuable experience. Key informants echoed these results, stating that while the workshop provided useful ideas, participating in Storytellers required students to think about their research in new ways, including considering audiences’ needs and determining the essential information required to support an understanding of research findings. The skills gained during Storytellers were said to be applicable throughout a participant’s career.

Increasing Awareness of SSHRC-Funded Research

Storytellers was generally perceived as having a modest impact on increasing awareness of SSHRC-funded research. While acknowledging that Storytellers gave entrants, particularly those who progressed to the Top 25, a wider audience with whom to share their findings, it was noted that this audience still tended to be members of the research community. Thus, participating in Storytellers gave entrants an audience outside of their specific institution or research discipline, there was limited reach into the general public. Although Storytellers did attempt to engage the mainstream media in the Challenge, the results of these efforts varied across years. As such, the bulk of its communication channels were geared toward the research community (e.g. SSHRC social media, institution write-ups).

Awareness and Promotion of Storytellers

There has been modest improvement in overall levels of awareness of the Storytellers Challenge among student and institutions. This is demonstrated by the increasing number of both students and institutions that have participated in the Challenge. SSHRC has increased its outreach efforts to eligible students to include social media, in addition to its direct marketing to students and institutions. These efforts resulted in over half (52%) of entrants reporting that they learned of Storytellers from SSHRC promotional efforts, as opposed to institutional promotional efforts (33%) or being told by a faculty member (23%).

Legitimization of Research Communication Skills

One of the key benefits of having SSHRC, a national research supporting organization, offer the Storytellers Challenge was that it legitimized the need for research communication skills. While there was growing acknowledgement that research communication skills were becoming more important, institutional efforts to promote them tended to utilize the traditional approach, focusing on supporting publication within specialty journals. As such, communication skills tended to be focused on reaching a small group of specialists. Storytellers both highlighted the importance of learning how to communicate to a broader audience and provided entrants with the opportunity to engage in and develop that ability. Entrants further noted that the prize offered by Storytellers helped to justify the time and effort required to develop the submission.

Effectiveness of Submission Judging

Numerous concerns were noted about the quality and transparency of how submissions were judged. Entrants appeared to be unclear about the judging process with “do not know” being the most common response when asked to rate the quality of the judging (32%) (satisfied with quality of judging=28%; dissatisfied with quality of judging=17%). Further, entrants reported being dissatisfied with the feedback they received on their submission (47%) and the explanation they received about why their submission was not selected for the Top 25 (71%).

Judges were more satisfied with the broader aspects of the judging process, with most reporting that they were provided with clear guidelines to judge submissions overall (81%). However, judges did note a lack of tools for specific aspects of judging. Specifically, 52% reported not having any training prior to judging, 52% reported that they did not have tools to take equity, diversity and inclusion (EDI) considerations into account when judging; and 34% reported not having clear guidelines to judge different submission types.

Concerns were also noted about the judging of the Top 25 at Congress. Some participants felt that a lack of French language skills by some judges hindered French presentations. It was also broadly noted that the quick pace of the presentations and judging impacted its quality, in part because judges may need to resort to gut ratings rather following the criteria.

Promotion of Equity, Diversity and Inclusion among Entrants

Although promotion of EDI is one of Storytellers’ goals, there are no clear, direct mechanisms to address or measure (EDI) in the Challenge. Minimal demographic information is collected from entrants and the Top 25, limiting the ability of the Challenge to determine its progress towards meeting its EDI goals. Interviewees who attended Congress provided anecdotal information suggesting diversity among the Top 25. However, they were unable to indicate if the diversity of the Top 25 was planned or incidental.

Efficiency

Many key informants reported that Storytellers had a broad impact with limited resources and budget. It was indicated that Storytellers generated greater media interest than other SSHRC initiatives (e.g. Impact Awards). It was thought that this was due to Storytellers’ efforts to share research findings more broadly. Additionally, Storytellers entrants generated materials that SSHRC could use to further disseminate research findings. However, despite this impact, some noted that Storytellers’ ability to evolve and adapt was hindered by its lack of dedicated resources. For example, the lack of dedicated staff limited the Challenge’s ability to review and adjust its operations. Additionally, the lack of clarity concerning the supports available to Storytellers impacted its ability to collect, store and analyze administrative data to better understand its progress since inception.

Recommendations

Given the wide recognition by all informants that Storytellers was filling an important role within the research community, it is anticipated that the Challenge will continue in the future. To ensure Storytellers’ continued impact, and help improve its reach, the following recommendations are provided. Some recommendations require additional resources, so SSHRC will need to review their impact in context with the expected scope of Storytellers. Further, a review of the objectives of Storytellers may be required to delineate the scope of the Challenge.

More robust administrative data collection and management

The lack of administrative data available to Storytellers impairs its ability to determine efficiencies and growth of the Challenge. Storytellers should maintain more consistent records of its operations, including annual budgets, formal agreements with partners, post-Congress debriefs, and so forth. Consistent administrative data will help Storytellers better understand its development over time. Assigning and clearly delineated administrative supports for Storytellers would help ensure that appropriate administrative data is collected, stored and analyzed.

Collect demographic information from entrants

The lack of demographic data prevents a deeper understanding of the reach of the Storytellers and hinders its ability to address EDI concerns. Entrant demographics would allow a better understanding of who is participating in Storytellers, as well as help inform targeted outreach efforts to under-represented groups.

Provide more comprehensive training and support for judges

While judges were satisfied with the broader criteria for judging submissions, they expressed concerns with specific aspects of it (e.g. EDI). Additional training and supports could address these concerns. For example, judges could also be provided unconscious bias training to explicitly address EDI concerns.

Provide feedback to all entrants

The lack of feedback provided to most entrants limits the learnings they take away from their participation. Providing all entrants with brief feedback (one page, table format) about their submission. would help students better understand the strengths and weaknesses of their submissions. This would enable students to learn from their submission to Storytellers, even if they were not successful. Additionally, providing feedback to all entrants could also focus and standardize the judging process, as judges would need to explicate their decisions.

Investigate reasons SSHRC-funded institutions do not participate

While an increasing number of institutions are participating in Storytellers, there are still many that do not. Reaching out to non-participating institutions would help identify barriers and plan future engagement efforts to broaden participation.

Consider offering separate prizes for submissions using different media formats

The general perception that video submissions were more likely to be selected for the Top 25 may impact the number of entrants willing to provide a text-based or audio submission. Given the technical knowledge and resources required to create a video submission, some potential participants may have opted not to enter Storytellers. Offering prize streams by submission format could encourage participation from entrants lacking the ability to create a video submission. Prize streams could also ease the judging process, with judges being assigned to specific format types.

1. Introduction

1.1 Background

The Social Sciences and Humanities Research Council (SSHRC) launched the Storytellers Challenge in 2013 to promote research and communicate the impacts and results of such research in the social sciences and humanities.

Storytellers, overseen by SSHRC’s Communication Division, is an annual Challenge that asks postsecondary students to show Canadians, in up to three minutes or 300 words, how SSHRC-funded research in the social sciences and humanities is impacting our lives, our world and our future for the better. Storytellers is open to postsecondary students enrolled in a Canadian postsecondary institution^{Footnote 1}. Entrants to Storytellers must provide a video, audio, text or infographic submission featuring a research project or initiative funded by SSHRC and carried out at the institution at which the Contestant is enrolled. Entrants are not limited to submissions based on their own research but can report on any SSHRC-funded research or initiative with the permission of the principal investigator.

Submissions to Storytellers are judged, with the Top 25 finalists receiving a $3,000 prize and the opportunity to present their submissions at the Congress of the Humanities and Social Sciences annual national conference. Of these 25 finalists, five winners are chosen to present their stories at the SSHRC’s annual Impact Awards Event in Ottawa. Both the finalists and winners are decided by a panel of judges that the Communications Division recruits on an annual basis. Judges include previous finalists, communications/public relations experts, representatives from non-profit organizations and think tanks, and other organizations.

The Challenge uses social and online media to advertise and promote its activities. Information about the Challenge is available on the SSHRC website in both English and French; this includes Challenge rules and regulations such as eligibility criteria and contest period, as well as entry, prizes, judging criteria, finalists, winners and general information.

1.2 Project Scope

This evaluation was conducted to provide SSHRC with an assessment of the Storytellers Challenge as it approaches its 10-year anniversary. The evaluation covered the period from 2012-2013 until the point of data collection in 2020-2021. The evaluation responded to questions in the areas of relevance and design, utilizing a previously developed evaluation approach.

The design incorporated multiple lines of evidence for each evaluation question; with data being collected through an administrative data and document review, key informant interviews, and surveys of entrants, judges, and participating institutions.

The results of the evaluation were designed to provide SSHRC with an understanding of issues such as the extent to which the Challenge was seen to address a key need; the level of awareness of the Challenge among the student population; the inclusivity of the Challenge; and participant satisfaction with the Challenge. The evaluation questions included:

Is there a continued need for the Challenge, with regard to its main goal (i.e., the promotion of the importance of SSHRC-funded research and its impact)?
Do the broader societal trends in terms of the consumption of information over the past decade (evolution of media and formats) affect the relevance of the Challenge? Has the Challenge capitalized on the growth and expansion in media (technology, capabilities, usage)?
To what extent are the targeted institutions and participants aware of and participating in the Challenge?
What do key stakeholders (entrants, finalists, postsecondary institutions, judges, Challenge and SSHRC management) identify as strengths and weaknesses of the Challenge?
Is the Challenge designed and delivered in a manner so as to encourage equity, diversity, and inclusion?
To what extent has the Challenge contributed to increasing awareness of the value and impact of SSHRC-funded research across Canada?
To what extent is the Challenge contributing to the communication of research results in the social sciences and humanities?
To what extent have participants gained valuable experience and enhanced their skills with regard to communicating the relevance and impacts of social science and humanities research?
To what extent is the design and delivery of the Challenge efficient?

1.3 Methodology

The evaluation collected multiple lines of evidence to answer the evaluation questions. The data collected and reviewed for the evaluation included:

Storytellers documents, including budgets, post-mortems, and entry guidelines;
Administrative data for Storytellers, including past participation numbers;
Documentation from other research communication competitions;
Key informant interviews with:
- Entrants;
- Judges;
- Workshop instructors;
- Participating institutions;
- SSHRC staff familiar with Storytellers; and
- Staff from the Congress of Social Sciences and Humanities; and
Surveys with:
- Entrants;
- Judges;
- Workshop instructors;
- Participating institutions.

Multiple lines of evidence allowed the evaluation to triangulate its findings to provide a more comprehensive analysis of Storytellers. A more detailed discussion of the methodology is appended.

2. Findings

2.1 Relevance

2.1.1 Role of Storytellers Within SSHRC

Storytellers fits within the mandate and objectives of SSHRC. Specifically, the mandate of SSHRC to promote and support postsecondary-based research and training in the social sciences and humanities. This mandate is fulfilled through the promotion of student entries showcasing SSHRC funded research. Promotion of student entries occurs at the Challenge level, as well as at the institution level, with institutions actively promoting the submissions of their students through local media channels (e.g. website, institution newspaper, local media releases).

Further, the goals of the Challenge align with SSHRC’s 2020–22 Strategic Plan objectives of growing Canada’s research talent and strengthening the research enterprise in Canada. Key informants stressed that Storytellers helps prepare students for future success in research by developing their skills in communication and knowledge mobilization.

2.1.2 Continuing Need for Storytellers

All key informant interviewees stated that they felt that Storytellers is meeting an important need within the research community. Storytellers both promotes the development of important researcher skills and improves the sharing of SSHRC-funded research results.

Research communication skills were viewed as important and increasingly essential for new researchers, as they allow for:

Democratization of knowledge – improved research communication ensured that knowledge was not restricted to a specialized few.
Better decision-making – sharing research allowed decision-makers and individuals to use research results to make better informed decisions.
Improved understanding – better research communication allowed the public to better understand the work being done, as well as its importance.
Accountability – sharing research results enabled the public to understand what tax dollars were being used to fund.
Improved action – clearer research communication can help people understand its importance in their lives (work with climate change and COVID-19 demonstrated this need).
Improved utilization– sharing research findings more broadly increases the possibility of communities/individuals/organizations incorporating or building off the results.
Increased career opportunities – new researchers may not end up in academia and have to know how to share their work with non-specialists (e.g. bosses, co-workers, funders).

Sharing of SSHRC-funded research results was also noted to be essential to:

Support SSHRC in the fulfilment of its mandate to share research findings.
Help ensure that Canadians were aware of the research their tax dollars were supporting.
Increase engagement of the public in research.
Help inform decision-makers and assist people in their daily lives.
Help develop the careers of researchers by promoting their work.

Continued interested in Storytellers was also illustrated through growth in the number of entries. Across the eight years of operation, the Challenge has seen a significant uptake, with a 285% increase in participation between 2013 and 2020. Participation in the Challenge increased on average 37% per year with the highest number of participants in 2020 with 335 entrants.

2.1.3 Alternates to Storytellers

Almost all key informants were aware of other research communication competitions. The most commonly mentioned alternate competition was Three-Minute Thesis (3MT), which was regularly hosted at institutions across Canada. However, informants stated that the alternate research communication competitions were more restrictive than Storytellers. It was stated that Storytellers had broader eligibility requirements, had fewer restrictions on the research entrants reported, and allowed a wider range of submission formats. For example, Storytellers was open to more students than any of the other competitions identified by participants. Most competitions were restricted to graduate students who were present on their own research projects. In contrast Storytellers was open to all postsecondary students who were only required to report on a SSHRC-funded project or initiative. Additionally, the format of other competitions was restricted to a single format, usually a spoken word presentation, unlike Storytellers, which allowed for audio, video, or text submissions, at the choice of the applicant. Interviewees regularly noted that the broader eligibility and submission requirements was one of the greatest strengths of Storytellers and helped to make it unique. It was further noted that the flexibility of submission formats encouraged creativity among applicants.

Table 2.1: Comparison of Research Competitions
Competition	Eligibility requirements	Research presented	Format requirements
The Storytellers	Canadian residents enrolled in a Canadian postsecondary institution	Research project or initiative funded by SSHRC	Three-minute video or audio submission or a 300-word text or infographic
Three-minute thesis	Graduate students attending a participating institution	Present their own research	Three-minute spoken word (no poems, songs) presentation with a single static PowerPoint slide (no additional electronic media)
GRADflix	Graduate students attending a participating institution	Present their own research	Sixty-second video
Ma thèse en 180 secondes	French speaking doctoral students enrolled in a participating institution	Present their own research	Three-minute oral presentation in French with a single static PowerPoint slide (no additional electronic media or props)
L’Institut d’histoire de l’Amérique française – Prix de l’Assemblée nationale du Québec	Best published work in political history	Original work published in French (or English if French translation published the same year)	Published material
Post Doc Research Slam	Current postdoctoral scholars at the University of Calgary	Present their own research	Three-minute spoken word (no poems, songs) presentation with a single static PowerPoint slide (no additional electronic media)

2.2 Effectiveness

2.2.1 Developing Participant Skills

Students entering Storytellers reported uneven benefits, with students who advanced to the Top 25 reporting benefits from participating in the Storytellers workshop. Instructors/Judges and Institutions were more likely to agree that the Storytellers experience supported improvements in students’ research communication skills than were entrants or students shortlisted for the Top 25.

Across all entrants, including those not shortlisted to the Top 25, benefits were generally limited to perceived improvements in participant research communication skills (Chart 2.2); with 55% reporting an improvement in research communication skills. Fewer students reported improvement in their overall communication skills (48%), an expanded network (23%) or improved academic connections (22%).

Chart 2.2 — **Source:** SSHRC Storytellers Entrant Survey Data Q.B18a, Q.B18b, Q.B18c, & Q.B18d

Entrants, selected for the Top 25, generally reported positive outcomes from participating in the Storytellers research communication workshop. (Chart 2.3). The Top 25 students felt the Workshop improved their research communication skills (63%); improved their ability to convey complex ideas (57%), helped them better understand the importance of research communication (53%), and improve their submission to Storytellers (50%).

Overall, the majority of the Top 25 students reported being satisfied with the workshop (62%) and felt that it was a valuable experience (61%) (Chart 5.8).

Chart 2.3 — **Source:** SSHRC Storytellers Entrant Survey Data Q.B11a, Q.B11b, Q.B11c, Q.B11d, & Q.B11e
N=56 (respondents who indicated not being selected for the Top 25 or Final 5 were not asked these questions)

The majority of the Instructors/Judges (87%) and Institutions (70%) surveyed stated that Storytellers enables students to improve their research communication skills. It is noteworthy that this group judged the benefits of participation more favorably than did the entrants.

Key informants also agreed that participation in Storytellers supported the improvement of research communication skills. Participation required students to think about their research in new ways, including considering the target audiences needs and the information that was required to support a conceptual understanding of the research. The skills gained were said to be applicable throughout a participant’s career.

2.2.2 Increasing Awareness of SSHRC-Funded Research

Storytellers had a modest impact on increasing awareness of SSHRC-funded research, in part because of the audience it is reaching. Key informants noted that the Top 25 presentations exposed research to audiences outside of a narrow discipline since Congress was attended by a broad range of researchers. The Challenge, however, was not thought to share the value and impact of SSHRC-funded research more broadly among the general public. This is primarily due the channels by which Congress and SSHRC distributed entrant’s presentations, which were focused on the research community, limiting the overall reach. Key informants noted that efforts to increase the reach of Storytellers to the broader public had only had sporadic results. Specifically, although Congress regularly made efforts to engage mainstream media with Storytellers, by putting reporters in touch with participants, there is varying uptake by the media, resulting in inconsistent coverage across the years. Further, there is limited viewing of the Top 25 submissions which are made available every year on the SSHRC YouTube channel. Only 33% of entrants felt that Storytellers increased the public’s exposure to SSHRC-funded research.

Not focusing on the public as the target audience, both Instructors/Judges and Institutions were slightly more optimistic that Storytellers helped increase awareness and value of SSHRC-funded research (Chart 2.2.3).

Chart 2.4 — **Source:** SSHRC Storytellers Instructors/Judges Survey Data Q.B10a & Q.B10b and SSHRC Storytellers Institutions Survey Data Q.B12a & Q.B12b

Institutions indicated that they actively promoted the submissions of local students, which expanded the reach of Storytellers. Promotion activities, typically focused on submissions selected for the Top 25, included:

Interviewing the student and preparing a story for the institutional newspaper;
Including information about, and links to, student submissions on the institution website;
Sending emails to other students about the winning submission;
Using social media to promote the story within the institution and local community; and
Preparing media releases about the winning submissions for local media.

More generally CSSH leadership and students noted that presentations generated through Storytellers were ready-made promotional materials given that they were designed for a non-specialist audience and were creative and engaging.

2.2.3 Awareness and Promotion of Storytellers

Over the years, Storytellers has made modest improvements in student and institutional awareness of the Challenge. The number of students (Chart 2.5) and institutions (Chart 2.6) participating in Storytellers has increased over time with the largest and most consistent growth beginning in 2018. Increases in participation have not occurred uniformly across both targeted language groups. French submissions have remained relatively constant across time while English submissions began increasing significantly in 2018 onward.

Chart 2.5 — **Source:** SSHRC Storytellers Administrative Data 2013-2020

Chart 2.6 — **Source:** SSHRC Storytellers Administrative Data 2013-2020

To support awareness of Storytellers, the Challenge provided information to institutional communication and research offices for distribution to appropriate students. As well, Storytellers provides Challenge information directly to SSHRC-funded researchers. Key informants stressed that while this communication model increases Challenge awareness it also biases outreach to universities, as colleges are generally listed as the recipients of grants, limiting information sent directly to college researchers.

In addition to direct outreach, key informants noted that Storytellers has also incorporated new technologies (i.e., social media) to extend its promotion and outreach efforts. Survey findings reflected these changes with more entrants reporting they had gained awareness of Storytellers through SSHRC outreach efforts, than local promotion by their institution (Table 2.7). While 52% of the entrants learned about Storytellers through SSHRC promotional activities, only 33% learned through the promotional activities of their postsecondary institution. In fact, communication directly to faculty members is almost as effective as promotion by the postsecondary institution in engaging students.

Table 2.7: Methods Entrants Learned of Storytellers
How did you become aware of Storytellers?	Response Count	Response Percentage
From SSHRC promotional activities	78	52%
From promotion by the postsecondary institution where I attended	49	33%
From a faculty member	35	23%
From competition alum (those who previously participated)	10	7%
From another student who had not previously participated	8	5%
From another source	6	4%
Do not know	3	2%
Total	151	100%
Source: SSHRC Storytellers Entrant Survey Data Q.B8 Note: Numbers may not sum to 100% due to multiple responses

Despite some postsecondary institutions supporting Storytellers promotion, half of the institutional representatives surveyed indicated not knowing if students from their institution were regularly participating in Storytellers (50%). Further, only one-quarter reported that students regularly participated (25%).

2.2.4 Legitimization of Research Communication Skills

Having a federal organization, such as SSHRC, run the Storytellers Challenge highlights the importance of communication skills for new researchers. Key informants noted that while postsecondary institutions are showing increased interest in research communication, many programs take a traditional approach to skill development, focusing on improving research communication by supporting publication in specialist journals. Similarly, entrants noted that postsecondary institutions focus on communicating research results only within a small group of specialists, with no emphasis on communication to a broader audience.

Key informants stressed that it is important to develop research communication skills in young researchers since it supports the democratization of knowledge. Researchers with strong communication skills are better able to broadly share research findings which provides the general public and decision-makers the opportunity to use this information to guide changes in their lives and communities. Further, being able to share research results with non-academics (e.g. funders) was viewed as increasingly necessary.

Storytellers has also spurred on the development of research communication skill development supports in some institutions. Key informants noted that these institutions have developed seminars to assist students entering Storytellers with their research communication skills.

Students also indicated that Storytellers prize amount helps to justify the time and effort required to submit an entry. In fact, some students interviewed noted that they probably would not have entered the Challenge if the prize were smaller. Overall, 63% of entrants were satisfied with the value of the prize.

2.2.5 Effectiveness of Submission Judging

Numerous concerns were noted about the quality and transparency of how submissions were judged. Entrants appeared to be unclear about the judging process, and many indicated they would have liked to have received more feedback on the strengths and weaknesses of their submission. While judges were satisfied with the broad guidelines, many expressed a need for more training and tools for judging. Further, interviewees felt that the Congress format did not support thoughtful or thorough judging of the Top 25.

Entrants were frequently ambivalent about the judging, with many indicating in the survey that they did not know how to rate its quality. Overall, survey responses tended to indicate a lack of knowledge about the process rather than dissatisfaction. Thus, while less than one-third of entrants (32%) agreed or strongly agreed that their submission was judged fairly, over half of entrants (62%) neither agreed nor disagreed (23%) or did not know (39%) if their submission was judged fairly.

As shown in Chart 2.8, a significant proportion of entrants were either neutral (neither satisfied nor dissatisfied) or did not know how to rate the:

clarity of shortlist selection criteria (34%).
shortlist selection process (41%).
quality of judging (55%).

Regardless of their understanding of the processes, entrants commonly expressed the need for more feedback from the judging process. Thus, 47% were dissatisfied with the feedback they had received on their submission and 71% were dissatisfied with the explanation about why their submission was not selected (Chart 2.2.7).

Chart 2.8 — **Source:** SSHRC Storytellers Entrant Survey Data Q.B10f, g, h, I, m, o

Judges were satisfied with the broader aspects of the judging process but noted a lack of tools for specific aspects of judging. The majority (81%) agreed that clear judging guidelines existed overall, however only 38% agreed that had clear guidelines to judge different submission types (Chart 2.9).

Similarly, while 57% of the judges agreed that they had tools to judge submissions, only 10% agreed they had tools to address EDI considerations (Chart 2.9).

Chart 2.9 — **Source:** SSHRC Storytellers Instructors/Judges Survey Data Q.8a,b,c,d
N=21 (Instructors/Judges survey respondents who indicated that they were not judges were not asked these questions)

Despite the availability of guidelines and tools, judges had not received sufficient training. Half of all judges (52%) reported not having any training prior to judging. Key informants more generally noted the need for increased training for Storytellers judges to ensure uniform application of the judging guidelines. Additionally, interviewees raised concerns about the in-person judging of the Top 25 at Congress.

Some participants noted that a lack of French language skills by some judges hindered French presentations. It was also broadly noted that the quick pace of the presentations and judging impacted the quality, in part because judges may need to resort to gut ratings rather following the criteria at this pace.

2.2.6 Promotion of Equity, Diversity and Inclusion among Entrants

Although promotion of EDI is one of Storytellers goals, there are no direct mechanisms to address or measure equity, diversity and inclusion in the Challenge. Currently, minimal demographic information is collected from entrants. When they enter the Challenge the only demographic information collected is official language. Gender and region of Canada are also collected from entrants selected for the Top 25. The lack of demographic information means it is not possible to determine the diversity of the entrants or measure the Challenge’s progress towards meeting its EDI goals.

Interviewees were unable to speak directly to the degree to which the Challenge was meeting its EDI goals. Some respondents provided anecdotal information about the diversity of the Top 25 which was felt to be diverse across gender, language, cultural background, and region of Canada. Key informants, however, stressed that diversity generally did not include Indigenous students.

Entrants also were generally unable to provide feedback on the degree to which the Challenge was encouraging diversity, many stating that they did not know if EDI was encouraged by the application criteria (27%), application process (29%) or the selection process (36%).

2.3 Efficiency

Key informants reported that Storytellers made a broad impact with limited resources and budget. In spite of sporadic outreach, key informants stated that Storytellers generated greater media interest than other SSHRC initiatives (e.g. Impact Awards). Informants thought this was because Storytellers took research out the confines of postsecondary institutions or specific research disciplines; Storytellers made the research results more broadly available. Additionally, Storytellers supports the generation of materials that SSHRC can use to further publicize the research they fund. Thus, Storytellers was felt to be a cost-efficient way to promote SSHRC funded research to a wider audience.

Despite the reach that Storytellers was having, key informants suggested that the lack of dedicated resources hindered its ability to evolve. More specifically, Storytellers operates on a limited budget with no dedicated staff assigned to manage the Challenge. The lack of staff has limited the Challenge’s ability to regularly review operations and plan for future development or changes.

The lack of resources has impacted Communications ability to collect, store and analyze sufficient administrative data to measure progress towards the Challenge’s outcomes. Key administrative data are unavailable for Storytellers, including budget figures across all years of operation, and annual Memorandums of Understanding with Congress. The lack of administrative data limits Storytellers ability to review and understand the progress achieved and growth experienced since the inception of the Challenge.

Further, the lack of certainty over where Storytellers is housed within SSHRC, impacts the resources it is able to draw upon. This was evidenced by the lack of clarity over which supports are available to Storytellers, as well as uncertainty about who should oversee administrative data or where that administrative data should be stored.

3. Recommendations

It was broadly reported among all stakeholders that Storytellers was filling an important role within the research community. Having SSHRC, a respected, national organization, provide Storytellers highlights the importance of research communication skills among new researchers. Study participants agreed that research communication skills were important and becoming increasingly essential for new researchers. Research communication skills allowed knowledge to be shared more broadly and provided researchers with invaluable skills for their careers. Both of these outcomes fit with SSHRC’s strategic priorities to:

Contribute innovative solutions to global challenges and opportunities by sharing research and demonstrating its value; and
Grow Canada’s research talent by developing skills in communication and knowledge mobilization.

Thus, through sponsoring the Storytellers Challenge, SSHRC:

Increased the dissemination of research findings;
Developed new researchers’ communication skills; and
Directly contributed to its Strategic Priorities.

To ensure Storytellers continued impact, and help improve its reach, the following recommendations are provided. Some of these recommendations require additional resources, so SSHRC will need to review their impact in context with the expected scope of Storytellers. Further, a review of the objectives of Storytellers may be required to delineate the scope of the Challenge.

3.1 More robust administrative data collection and management

The lack of administrative data available to Storytellers impairs its ability to determine efficiencies and growth of the Challenge. Without year-over-year data being collected, it is difficult to determine how Storytellers is changing and evolving over time. To address this, Storytellers should maintain more consistent records of its operations, including annual budgets, formal agreements with partners, post-Congress debriefs, etc. Consistent administrative data will help Storytellers better understand its development.

Part of the challenge of managing Storytellers administrative data lays with the uncertainty of the Challenge’s position within SSHRC. The lack of clarity over the supports available to manage Storytellers creates confusion about where its administrative data should be stored and analysed. Assigning and clearly delineating administrative supports for Storytellers would help ensure that its history, evolution and adaptations are better understood and would promote better planning for its future.

3.2 Collect demographic information from entrants

The lack of demographic data prevents a deeper understanding of the reach of the Storytellers. Additionally, it hinders Storytellers ability to determine its ability to address equity, diversity and inclusion in the Challenge. Collecting demographic information from entrants would allow SSHRC to better understand who is, and who is not, participating in the Challenge, as well as its success in addressing EDI concerns. Further, demographic data would help SSHRC determine if and how targeted outreach and promotion efforts could be used to be expanded to ensure inclusivity.

To address potential privacy concerns from entrants, demographic information could be collected on a voluntary basis. Entrants should also be provided with a clear reason for why the information is being collected and how it will be used. Finally, the demographic information should be kept separate from materials provided to judges to reduce bias and, potentially, could be kept entirely separate from submissions.

3.3 Provide more comprehensive training and support for judges

While satisfied with the broader criteria for judging, there were concerns with specific aspects of it (e.g. EDI). These concerns could be addressed with the provision of additional training and supports. For example, a more detailed and comprehensive judging criteria could be provided to judges. This criteria could include examples for each different format, highlighting their various strengths, to provide better guidance when comparing formats. Judges could also be provided unconscious bias training to explicitly address EDI concerns.

3.4 Provide feedback to all entrants

Currently Storytellers provides no feedback for entrants not selected in the Top 25. As a result, there is limited learnings that most entrants take away from their participation in Storytellers. This is further impacted by the lack of transparency in the judging process. Entrants who are not selected for the Top 25 both receive no feedback about their submission and are not clear why their submission did not advance.

Providing all entrants with brief feedback (one-page, table format) about why their submission was, or was not, selected for the Top 25 would help students better understand the strengths and weaknesses of their submissions. This would enable students to learn from their submission to Storytellers, even if they were not successful. Additionally, providing feedback to all entrants could also focus and standardized judging process, as judges would need to justify their decisions.

3.5 Investigate reasons SSHRC-funded institutions do not participate

While an increasing number of institutions are participating in Storytellers, there are still many that do not. Targeted outreach to non-participating institutions could help determine potential barriers that may limit their participation (e.g. lack of awareness, lack of resources). Knowledge of these barriers would help Storytellers plan future engagement efforts to ensure broad participation from all institutions.

3.6 Consider offering separate prizes for submissions using different media formats

There was a general perception that video submissions were more likely to be selected for the Top 25 and Final Five. As a result of this perception, potential participants may opt not to participate in Storytellers, feeling they lack the necessary skills and/or resources to create a video. Without a video, they could perceive the time and effort to create another submission type would be waste of effort, as they were not likely to win.

This perception could be countered if there were multiple prize streams, for each format type. Having different prizes for different types would ensure that all format types are represented in the Top25. This could help to encourage participation from individuals lacking the confidence to make a video. Entrants could select the submission format that best meets their research results and the strengths, rather than a format that would increase their chance of winning.

Having multiple prize streams could also allow for streamlining the judging process. Judges could be assigned to specific format types, reducing the challenges of judging between types.

Appendix A—Methodology

The methodology used during the evaluation included data collection through an administrative and document review, key informant interviews and surveying.

Document Review

A document review was conducted to develop an in-depth understanding of the context in which Storytellers operates, its position in relation to other research funding programs/initiatives with similar objectives, and its alignment with current federal priorities. Relevant documentation and administrative data were provided by SSHRC, which was reviewed against the evaluation questions. Storytellers provided the following documentation and data:

Challenge background documentation;
SSHRC’s 2016–20 Strategic Plan;
SSHRC’s 2020–2022 Strategic Plan;
Administrative data on entrants, finalists, and winners, including:

Application data; and
Award data.

Information on Storyteller Challenge budget;
Information on communication strategies and activities;
Participant guidelines and entry criteria; and
Challenge “post-mortem” PowerPoint presentations.

Additionally, documentation on similar competitions was obtained using publicly available search tools, academic literature databases, and grey literature databases. Findings from the document and administrative data review were summarized, by evaluation question, in a technical report.

Key Informant Interviews

The stakeholders interviewed were involved with Storytellers in different capacities, generating a comprehensive understanding of the Challenge. Semi-structured interviews were conducted with interview guides that were prepared for each stakeholder group. The guides ensured that the same information was collected from all participants, while allowing individuals to focus on topics they felt were most important.

Interviews were conducted from September 3 to October 13, 2021. Interviews took between 30 minutes to one hour and were all completed over the phone in the official language preferred by the participant. Interviewers recorded all interviews, with the permission of the respondent, and took notes during the discussion. Audio recording were used to review interview notes and ensure their comprehensiveness.

Analysis of the interviews was conducted by stakeholder group. A thematic analysis was conducted to extract common perceptions about Storytellers, as well as capturing differing and unique perspectives.

In total, 29 interviews conducted with stakeholders, as shown in the Table 1 below.

Table 1.1: Number of Interviews by Stakeholder Group
Stakeholder Group	Number Invited	Number of Interviews
Entrants	107*	11
Institutions	60**	7
Instructors/Judges	31***	3
Congress for Social Sciences and Humanities (CSSH) Leadership	4	4
Challenge Staff	4	4
Total	206	29
* 89 invitations resulted in no response; 7 resulted in refusals 46 invitations resulted in no response; 7 resulted is refusals. * 23 invitations resulted in no response; 5 resulted in refusals.

Entrants

There were 11 interviews with entrants to Storytellers. Nine of the entrants interviewed progressed into the Top 25; five of which were in the Final Five for their application year. Only two of the 11 entrant interviewees reported that they did not make it into the Top 25. All but one of the interviewees participated in Storytellers for only one year; one participated in Storytellers twice. Two of the entrant interviewees also reported later being a Challenge judge.

Institutions

There were seven interviews with postsecondary institutions. Most of the institution interviews were conducted with individuals from a Research Office (n=4) (e.g. Director of Research); two were conducted with Communication staff (e.g. Director of Communications); and one was conducted with a professional development coordinator, whose work was directed toward graduate students. Most of the interviews (n=5) were completed with universities, and the remaining two were completed with colleges. One of the participants reported being responsible for the development of a workshop to assist students who were interested in participating in Storytellers.

Instructors/Judges

There were three interviews with workshop Instructors/Judges. Two of the Instructors/Judges reported being involved in Storytellers for multiple years, both as an instructor and as a judge. The other interview participant reported acting as a Storytellers’ judge for a single year.

CSSH Leadership

There were four interviews with CSSH Leadership. Participants in the CSSH Leadership interviews indicated a range of involvement with Storytellers. Most (n=3) reported very indirect involvement (oversaw the event team that managed Storytellers, promoted Storytellers within Congress). One participant reported working closely with SSHRC on the management of Storytellers.

Challenge Staff

There were four interviews with SSHRC Storytellers Staff. All interviewees reported being involved with Storytellers for multiple years. Two of the participants reported being involved in the development and original implementation of Storytellers. The range of involvement included overseeing Communications where Storytellers is housed, to organizing and coordinating the execution of Storytellers.

Surveys

The survey instruments were developed in conjunction with SSHRC and allowed Entrants, Institutions, and Instructors/Judges to provide feedback on their experiences with Storytellers. Full survey administration occurred from September 13 to October 12, 2021.

A sample file was provided by SSHRC of Entrants, Institutions, and Instructors/Judges who participated in Storytellers from 2013 to 2020. Individuals were sent an email inviting them to complete the Storytellers survey online. Respondents who did not complete the survey were sent up to 7 reminder emails (respondents could opt out of receiving further email reminders).

Table 2: Number of Surveys by Stakeholder Group
Stakeholder Groups	Number of Individuals Invited to Participate	Number of Survey Completions
Entrants	1,278	151
Institutions	440	23
Instructors/Judges	93	20
Total	1,811	194

Limitations

The following limitations of the data collection methods should be kept in mind when interpreting the data.

The survey sample included individuals who participated in Storytellers over the entirety of its administration; from 2013 to 2021. Individuals who participated in Storytellers earlier may have had outdated contact information, may have been less likely to recall details of their involvement with Storytellers, and, as such, may have been less likely to complete the survey. Therefore, participation may be biased toward individuals who were involved with Storytellers more recently.
Due to the internal IT policies relating to the email settings at the University of British Columbia and the University of Waterloo, invitations to students with these email domains (ubc.ca, uwaterloo.ca) were rejected by their servers. Overall, only 17 (out of 80) UBC and 4 (out of 41) University of Waterloo students were sent invitations because they had alternate emails.
Participation in the survey or in the key informant interviews may also be affected by the level of involvement individuals had with Storytellers. Individuals with greater involvement, such as being selected for the Top 25 or Final Five are more likely to both remember participating in the Storytellers, and recall more details about the Challenge and thus agree to participate in either the survey or a key informant interview.
Many of the key informants interviewed did not have detailed knowledge of, or experience with, Storytellers which may have resulted in top-of-mind responses to questions. Although guides were provided in advance to help address this, detailed review of the guides in advance is typically rare. This led to many of the interviewees providing their perceptions and opinions about the operations and impact of Storytellers, with limited examples and information to support their perspectives (some respondents explicitly acknowledged this limitation).

Date modified:: 2023-05-30