Evaluation of Awards to Scholarly Publications

Evaluation of Awards to Scholarly Publications
December 2020

(PDF, 758 KB)

Produced by the SSHRC Evaluation Division
Ference & Company Consulting Ltd.

The Honourable François-Philippe Champagne, P.C., M.P.
Minister of Innovation, Science and Industry

© Her Majesty the Queen in Right of Canada,
represented by the Minister of Industry, 2021

Cat. No. CR22-120/2021E-PDF
ISSN 978-0-660-38959-2

List of Acronyms

• ASP – Awards to Scholarly Publications
• ASPP – Awards to Scholarly Publications Program
• CCA – Canada Council for the Arts
• CBF – Canada Book Fund
• CH – Canadian Heritage
• EAC – Evaluation Advisory Committee
• DORA ‒ Declaration on Research Assessment
• ECR - Early career researcher
• EDI - Equity, diversity and inclusion
• FHSS - Federation for the Humanities and Social Sciences
• OA - Open access
• SSH – Social Sciences and Humanities
• SSHRC - Social Sciences and Humanities Research Council

Glossary of Terms

Canadian research

For the purpose of this evaluation, Canadian research in the context of ASP has been defined as research conducted by researchers affiliated with Canadian institutions or about Canadian topics (for example, on topics of Canadian geography, culture, history or institutions).

Canadian scholarly publisher

In the context of ASP, a Canadian publisher is at least 75% owned and controlled by Canadians and has headquarters and at least 50% of staff in Canada.

Early career researcher (ECR)

Academic researchers with five years’ experience or less since their first academic appointment, with considerations for part-time positions and interruptions.

Equity, diversity and inclusion (EDI)

A tri-agency initiative to increase equity in the workplace for equity-seeking groups. This includes the following four designated groups: women, visible minorities, Indigenous Peoples, and people with disabilities.

Knowledge mobilization (KMb)

SSHRC defines knowledge mobilization as follows: “the reciprocal and complementary flow and uptake of research knowledge between researchers, knowledge brokers and knowledge users—both within and beyond academia—in such a way that may benefit users and create positive impacts within Canada and/or internationally, and, ultimately, has the potential to enhance the profile, reach and impact of social sciences and humanities research” (SSHRC, 2018).

Open access (OA)

Open access is defined as availability of the publication online and without charge to the reader.

Research dissemination

Efforts to enhance accessibility and awareness of research findings among potential users. Research dissemination is a component of KMb, and concerns activities within the supply side of a KMb system, distinct from uptake or utilization by others. Dissemination is also understood to involve movement of information, whereas KMb is more broadly concerned with knowledge (for example, including implementation in practice)^{Footnote 1}

Scholarly book

An extended or long-form academic text that demonstrates quality scholarship and is deemed to represent a contribution to knowledge and research. Books eligible for ASP funding include monographs, collective works, critical editions, critical bibliographies, reference works and documentary collections. Non-eligible submissions include unrevised theses, works of fiction and textbooks, among others (see Appendix A).

Acknowledgement

We would like to thank the Canadian SSH researchers, publishers and other sector stakeholders who contributed their time as participants in this evaluation.

We would also like to thank the external members of the Evaluation Advisory Committee for their expert advice and support: Isabelle Bourgeois, PhD, and Stefanie Haustein, PhD, from the University of Ottawa, and Josée Dallaire, Sonia Vani and Gina Hill Birriel from the Federation for the Humanities and Social Sciences.

Executive Summary

About Awards to Scholarly Publications

Awards to Scholarly Publications (ASP) is funded by the Social Sciences and Humanities Research Council (SSHRC) and administered by the Federation for the Humanities and Social Sciences (FHSS). It is designed to support the dissemination of Canadian social sciences and humanities (SSH) research by funding the publication of high-quality scholarly monographs, books and other long-form publications. ASP currently offers approximately 180 Publication Grants and five Translation Grants each year for a total annual contribution of $1.5M dollars.

About the evaluation

This evaluation was conducted as a joint evaluation of two funding opportunities: Awards to Scholarly Publications (ASP) and Aid to Scholarly Journals (ASJ). This report covers ASP.

Program relevance and performance were identified as priority areas for evaluation, including how the design of ASP and ASJ contributes to their performance. Cost-efficiency was also covered as a usual component of SSHRC evaluations. The evaluation’s overall purpose was to inform decisions by senior and program management about the future of ASJ and ASP in a changing environment.

The evaluation was led by the SSHRC Evaluation Division in collaboration with Ference & Company, an external consultancy. It was guided by an Evaluation Advisory Committee (EAC)^{Footnote 2} with oversight by the SSHRC Departmental Evaluation Committee (DEC)^{Footnote 3}.

Conclusions and recommendations

Relevance

ASP is relevant to SSH researchers. However, its relevance is highly concentrated in specific areas of the SSH research community. ASP is also highly relevant to Canadian scholarly publishers. In particular, ASP makes an important contribution to a subset of university presses. There is no equivalent to ASP funding among other federal programs.

ASP’s objectives align generally with federal priorities in terms of research excellence and dissemination. However, these objectives are very broadly stated. In practice, the relevance of ASP rests on the alignment between SSHRC and scholarly publisher mandates to support Canadian SSH research. Alignment is evident between ASP and specific SSHRC priorities, such as support for first-time authors and research on Canadian topics. A number of publishers report EDI-related mandates.

Performance

The findings indicate that ASP is achieving its primary intended objective to support research dissemination by contributing to the amount of SSH research published. ASP’s support enables scholarly publishers to invest in manuscripts of high scholarly value but low cost-recovery potential.

ASP has no mechanism to contribute directly or substantively to the quality of published research, although it upholds a consistent standard for peer review. As quality processes are already embedded in research and scholarly publication practices, ASP may be best positioned to contribute through indirect support, that is, by directing funding to where quality standards are high. There is some limited evidence of a small, secondary contribution of ASP in other areas, for example increased publisher capacity for marketing and promotion. Accessibility and discoverability aligned with digital and “open” publication are not currently targeted by ASP.

Cost efficiency

ASP’s cost efficiency ratio is high with operating costs at 22¢ per $1 in grant funds awarded. This is due to the funding delivery mechanism. Improving ASP cost efficiency in a substantial way would require changes to the delivery mechanism.

Alternatives

Both strengths and weaknesses were reported for ASP’s funding model. As examples: the current model avoids incentivizing quantity over quality and offers flexibility in use of funds, which is important given the diversity in SSH books and in the publishing sector. However, the process delays publication of manuscripts and provides minimal flexibility for SSHRC to target funding to priority areas.

Recommendation 1. Continue to offer support for long-form publishing of Canadian SSH research. The funding fills a niche not addressed by other funding, is relevant to Canadian researchers, and provides capacity for the publication of research that is important for SSH and Canada.

Recommendation 2. Develop more fully articulated and concrete objectives for ASP. ASP’s objectives are broad and ambitious for its small size and the concentrated relevance of the funding for SSH research. A set of more concrete objectives is needed for program management and SSHRC to situate and guide the funding opportunity. Specific suggestions are provided in Section 6.

Recommendation 3. Identify options to update ASP’s funding mechanism. SSHRC is currently working collaboratively with FHSS and the Association of Canadian University Presses (ACUP) to develop open access (OA) capacity in scholarly book publishing and to increase minority language participation in ASP. In concert with those efforts, SSHRC should explore alternate delivery mechanisms for ASP to improve cost efficiency and performance. Suggestions are provided in Section 6.

Recommendation 4. Update ASP’s program logic model or theory of change model. A program logic model or theory of change should be drafted for ASP. The findings reported above provide the basis for an empirically based change model. This would be useful as a baseline for development going forward.

1.0 Introduction

This report presents key findings, conclusions and recommendations from the evaluation of the Awards to Scholarly Publications (ASP) funding opportunity conducted in 2019-20.

1.1 Evaluation background and purpose

About ASP. Awards to Scholarly Publication Program (ASPP) is funded by SSHRC and administered by the Federation for the Humanities and Social Sciences (FHSS). It is designed to support the dissemination of Canadian SSH research by funding the publication of high-quality scholarly monographs, books and other long-form publications. ASP has been active for 79 years. It currently offers 180 Publication Grants and five Translation Grants each year for a total annual contribution of $1.5M dollars. A more detailed description of ASP is available in Appendix A.

About the evaluation. This evaluation was conducted as a joint evaluation of two funding opportunities: Awards to Scholarly Publications (ASP) and Aid to Scholarly Journals (ASJ). This report covers ASP.

Program relevance and performance were identified as priority areas for evaluation, including how the design of ASJ and ASP contributes to their performance. Cost efficiency was also covered as a usual component of SSHRC evaluations. The evaluation’s overall purpose was to inform decisions by senior and program management about the future of ASJ and ASP in a changing environment.

The evaluation was led by the SSHRC Evaluation Division in collaboration with Ference & Company, an external consultancy. It was guided by an Evaluation Advisory Committee (EAC)^{Footnote 4} with oversight by the SSHRC Departmental Evaluation Committee (DEC)^{Footnote 5}.

Evaluation questions and scope. The evaluation focused on five questions:

Question 3 has a dissemination component. This component focused on accessibility/discoverability of publications as related to activities of funding recipients. The choice of focus was due to the structure and small size of ASP. Indirect outcomes related to reader behaviour (e.g., citations) were outside the evaluation’s scope. Question 5 focused on alternative examples of government funding for OA initiatives. OA was chosen as the focus because advancing OA publication is a current key priority for SSHRC. For ASP, some attention was paid to strengths and weaknesses of the current funding model because of the time since ASP’s last evaluation (2004) and interim changes in the context of scholarly publishing. The evaluation covered the period 2008-17 for performance^{Footnote 6} and 2008-19 for questions of relevance. Cost efficiency covered the most recent four fiscal years (2014-19).

Evaluation design. A two-phase sequential mixed methods design was used, under a broad utilization-focused (Patton, 2008) and theory-based (Chen, 2006) evaluation approach. Phase 1 was designed to establish a foundational understanding of the potential influence and relevance of ASP, including development of a draft theory of change. This phase was needed because of changes to ASP in the time since the last evaluation, as well as changes in the surrounding context and an absence of pre-program baseline data. Phase 2 was designed to test and refine Phase 1 results to arrive at final findings.

Methods. Phase 1 used the following lines of evidence:

• key informant interviews (n=41) with program management, publishers, researchers and sector stakeholders, with 25 of these interviews related directly to ASP;
• longitudinal analysis of ASP administrative data (2008-17); and,
• reviews of academic literature, working papers and program documentation, including two recent studies about ASP (Fast, 2018; Lobet & Larivière, 2018).

In Phase 2, the evaluation used the following lines of evidence:

• a survey of ASP-eligible publishers (n=42) and SSH researchers (n=993);
• focused analyses of ASP administrative data (2008-18) and SSHRC administrative data, including financial data;
• focused analyses of publication data for SSH researchers (n=1306); and,
• focused analyses of external website data on publications (n=120).

A detailed description of methods is available in Appendix C.

Methodological strengths and limitations. The principal advantage of mixed methods designs is the ability to leverage strengths of both qualitative and quantitative methods. For example, in this evaluation, limits to generalizability of findings from interviews and limits to case-sensitivity of survey data are offset through synchronization. Important limitations exist, however, primarily related to available data sources. No comparison group or pre-program data are available for ASP. Although the evaluation triangulated data from different sources on each indicator, results for some indicators are largely reliant on self-report data (e.g., through survey and interviews) because directly comparable administrative or externally sourced data were not always available. In such cases, results were compared across multiple questions and respondent subgroups where possible to increase confidence. However, caution is recommended in the interpretation given the likelihood of systematic response bias.

Lastly, it is important to note that the Canadian scholarly publisher and SSH researcher populations are heterogeneous. For example, SSH has numerous disciplines with their own scholarly publication practices, as well as regional and linguistic differences. The ASP-eligible publisher population is made up of specialist, generalist and university presses, wide variation in size of presses, and presses that focus in either or both official languages. Aggregate results (i.e., overall averages) may obscure important variation among subgroups. Subgroup-level analysis was conducted where possible (e.g., among language and disciplinary groups) but was limited by the small size of some subgroups in the samples. Results should not be considered generalizable outside the populations defined for the study (see Appendix B).

1.2 Structure of this report

Section 2 presents findings on relevance, Section 3 on performance, Section 4 on cost efficiency and Section 5 on alternatives. Key conclusions and recommendations follow in Section 6. A profile of ASP and details on methods and data sources are available in the appendices.

2.0 Relevance

The evaluation assessed relevance in terms of importance to Canadian SSH researchers, the extent to which ASP is critical to scholarly book publisher operations, and alignment of ASP’s objectives with federal roles and priorities.

2.1 Conclusions

ASP is relevant to SSH researchers. However, the importance of this funding is concentrated in specific disciplines and areas of research. ASP is also relevant to researchers for whom collaboration with a Canadian scholarly publisher can add value (e.g., first time authors, authors writing in French, researchers working on topics that require Canadian contextual knowledge).

ASP is highly relevant to Canadian scholarly publishers, especially university presses. ASP contributes to publisher capacity to invest in books that are of high scholarly value but have limited potential for cost recovery (i.e., due to a specialized or national audience). There is no equivalent to ASP funding among other federal programs, and other federal funding does not incentivize scholarly publication.

Alignment is evident between ASP and SSHRC priorities. ASP’s objectives align generally with federal priorities in terms of research excellence and dissemination. However, these objectives are very broadly stated^{Footnote 7}. In practice, the relevance of ASP rests on alignment between SSHRC and scholarly publisher mandates to support Canadian SSH research. This is because ASP pursues its objectives via the work of publishers. Although ASP is explicitly a program for researchers (as an award for authors), more than 90% of applications are developed and submitted by publishers: the publisher selects which manuscripts to submit, coordinates peer review, invests in editorial development of the manuscript before application to ASP, submits the application, and receives and manages the awarded funds.  

ASP’s objectives do not yet incorporate EDI. ASP collects monitoring data in related areas: participation by first-time authors and French-language publications. The data indicate ASP is important for first-time authors, but challenges exist related to systemic barriers for francophone authors.

2.2 Detailed findings on relevance

Question 1. Is there a need for the federal government to provide direct financial support to scholarly book publishers to increase dissemination of Canadian SSH research?

Relevance of book publishing for Canadian SSH researchers

Researchers surveyed for this evaluation were asked how important scholarly books are in terms of various aspects of their academic work. Respondents in the humanities (n=69) provided substantially higher importance ratings in all areas than researchers in social sciences (n=183). Responses from those working across both social sciences and humanities (n=29) tended to fall in between (Figure 1, below).

A majority of respondents reported that scholarly books are absolutely essential or very important to them as source material (for teaching and their research). In terms of importance of books for publishing their own research or for their career progression, however, ratings by humanities researchers remained high while social sciences researcher ratings were mixed: about one-third reported books to be absolutely essential or very important and about one-third reported books to be of little or no importance.

Figure 1. Importance of books for aspects of teaching and research, humanities versus social sciences.

Survey data were analyzed at disciplinary level for two areas: publication of research and career progression because of the variance in response. Among the disciplines with sufficient representation in the survey (n>20), Figures 2 and 3 illustrate much higher importance ratings by researchers in history, languages/literature, arts and political science than in psychology and economics/business.

Figure 2. Importance of books for publication of own research

Figure 3. Importance of books for career progression

ASP submissions data are consistent with the above. History, anthropology/cultural studies and political science/political economy are the top disciplines of manuscripts submitted to ASP between 2008-17, followed by literature, then sociology/social work. Economics and psychology appear near the bottom of a ranked list by number of submissions.

Researchers who rated scholarly books as important in terms of publication of their research findings were asked in the survey to describe the way(s) in which they are important. Close to three-quarters of respondents (72.3%, n=227) answered this question.

“Books are a very different genre than journal articles and allow for expressing and communicating research in depth... Articles in contrast are short and temporal interventions reflecting the present moment.” (Mid-career academic)

The most common response (42%) was that book-length format is necessary to develop the content, for example, to develop complex ideas or provide depth necessary to a topic. In some disciplines, book-length format is a norm and aligns best with how research projects are structured. Book-length format was also described as important to new areas of research or new forms of analysis. Several respondents also identified books as better than journal articles to reach non-academic or interdisciplinary audiences and students.

The importance of Canadian publishers to SSH researchers. Based on analysis of data in the SSHRC Canadian Common CCV database (n=1306)^{Footnote 8}, an estimated 31% of Canadian SSH researchers have at least one book publication, and 17% have published with a Canadian book publisher (95% CI [14.3 to 19.1]). This equates to approximately 4,680 of an estimated 28,000 Canadian SSH researchers^{Footnote 9}. For comparison, including co-authors, a total of 3,487 authors appear on applications to ASP in the decade 2008-17.

Among respondents to the researcher survey (n=314)^{Footnote 10}, significantly more humanities researchers (73%) reported they had published a book with a Canadian scholarly publisher compared with social sciences researchers (46%), X2(6, 314), p<.001. Slightly more French-speaking respondents (58%) reported publishing with a Canadian publisher than English-speaking respondents (50%), but the difference was not statistically significant. Responses were similar among respondents by career stage (at the time of publication) and by gender.

Researchers responding to the survey who had published a book with a Canadian publisher (n=163) were asked what characteristic was most important to them when selecting a book publisher: 153 researchers responded (94%). The top three characteristics were the same across language and career stage groups:

“There are so many book publishers in my field that are predatory, or that do not conduct proper or any peer review… despite being very prolific.” (Early career academic)

1. Reputation. Just under half (44%) described reputation as important because the publisher’s reputation lends credibility to the book. Several stated that reputation indicates the quality to be expected from the production process, and a few noted the rise of predatory publishing.
2. Content, values and audience. More than one-quarter of respondents (28%) identified a good fit between their research and the type of content the publisher carries, its mandate and the audience it serves. This helps to ensure the editorial team is knowledgeable, that peer reviewers will be selected appropriately, the book will reach the audience it is meant for, and because it increases likelihood of acceptance. A handful of respondents referred to the mandate of the press and its experience with research involving special populations, e.g., in the context of Indigenous research, a commitment by the press to reconciliation and anti-racism.
3. Editorial quality, working relationship with authors. One-quarter of respondents (25%) indicated that the quality of the editorial team and the strength of its working relationship with authors makes a substantial contribution to the manuscript, explaining that publishing a book is an enormous amount of work and can be even more challenging without a strong relationship.

Canadian publishers. These researchers were also asked what characteristic was most important to them when selecting the publisher for their most recent Canadian publication. The most frequently identified characteristics were similar to the above and similar across career stage and language groups, although French-speaking respondents also identified the language of the press with equal frequency as quality of the press.

When asked if they would have attempted to publish with a non-Canadian publisher if they had been unable to publish in Canada, 63% replied “yes” (n=102), 18% replied “no” (n=30) and 19% replied “don’t know”. Researchers who replied “yes” (they would have attempted to publish outside Canada) were asked if doing so would have changed anything about the book project. Half replied “no” (50%) and 27% replied “yes” (n=28)^{Footnote 11}. Those who replied “yes” reported a need for content changes e.g., less Canadian-specific content or a need to write in English instead of French.

“The content was Canadian―it would have been a lot of work to make it US-centric enough for a US publisher.” (Established academic)

Among those who replied that they would not seek to publish outside Canada (18%, n=30), the most common reason was that the topic was Canadian (for example, “la littérature québécoise”) or intended for a Canadian audience. Many of these respondents stated that a Canadian press could give this content the best support or that it was unlikely to be of interest to a non-Canadian publisher. In addition to Canadian topics, examples were provided of topics in emerging or controversial research areas.

“We submitted our prospectus to several American publishers, with no luck despite excellent peer reviews…In a Canadian academic press, we find a champion for a topic that is most definitely of interest to researchers and professionals in our field across North America, but which US presses may find to be off-limits. A Canadian press is vital for my research area.” (Early career academic)

Overall, 36% (n=58) of survey respondents who have published with a Canadian book publisher reported that there would have been an adverse impact on their book had they not been able to publish in Canada.

In sum, the relevance of ASP is concentrated in specific areas of the SSH research community. Long-form publication is very important in the humanities but mixed in social sciences. Overall, the estimated proportion of SSH researchers who have published with a Canadian book publisher is less than 20%. Among those surveyed, just over one-third describe adverse impact were they unable to publish in Canada.

Relevance for publishers: importance of ASP for viability of book publisher operations

The ASP award represents an estimated 40-47% of the cost to publish a scholarly book (on average). This estimate is based on publisher responses to the survey for this evaluation (n=19),^{Footnote 12} and is consistent with other data from a recent study on ASP (Fast, 2018). This amount represents a substantial proportion of cost to offset through other means if ASP were not available.^{Footnote 13}

“If we didn’t have [ASP], would we go bankrupt? I guess the answer is probably not, but the press would be very, very different from what it is today, in terms of the number of titles published, in terms of the types of titles published, in terms of the standard of marketing that we do, possibly in terms of the staffing that we employ.” (University press representative)

Findings from multiple lines of evidence^{Footnote 14} are consistent: while ASP is not reported to be essential to the immediate survival of most scholarly presses, without ASP publishers would need to make different decisions about what they publish to maintain their viability. ASP is reported to be particularly important to publisher capacity to support books in their publishing program that are high-cost to produce and/or have low cost-recovery potential. Differential contribution of ASP to specific types of authors and types of books is described further below.

When publishers were asked in the survey for the most significant impact of a loss of ASP, all respondents to this question (n=18) projected the most significant impact to be fewer scholarly books published by their press. Survival was mentioned by one respondent who described the press potentially closing if the loss of ASP had a cascading effect in other areas (including other funding). In key informant interviews with publishers and sector stakeholders, all respondents similarly stated that the most significant impact of the loss of ASP would be reduced capacity to publish scholarly books (n=14). Secondary effects reported by interviewees include diminished reputation of the press, less ability to accept risk and a need to recover costs through other means (e.g., higher price point).

In the survey, publishers were also asked to estimate the impact of the loss of ASP funding across areas of their operations. Their responses are depicted in Figure 4, below. As shown, 68% estimated a major impact on the number of books undertaken each year. About half (53%) projected a major impact on marketing and promotion. A substantial number (42%) projected a major impact on their ability to make progress toward strategic goals and to respond to changing conditions in the sector. In contrast, quality assurance, editorial support provided to authors and digital publication were areas of least projected impact although these related directly to the number of books undertaken, e.g., editorial support may remain at similar levels for fewer books.

Figure 4. Publisher estimates of ASP impact in areas of operations (n=19)

Differential importance among publishers. 43 publishers participated in ASP between 2008-17.^{Footnote 15} However, 10 presses submitted 93% of ASP applications and received 93% of ASP funds.^{Footnote 16} Based on awards granted in the most recent three fiscal years (2015-19), these 10 presses can be divided into two groups:

• Group 1. Three university presses. These presses received 71% of ASP grant funds in this period, averaging $351,111 per year each (range $296,000 ‒ $410,667).
• Group 2. Seven university presses. These presses received regular but less substantial funding, averaging $46,095 per year each (range $18,667 ‒ $117,333/year). Two of the presses in Group 2 are associated with francophone universities.

A third group of 33 publishers has collectively received only 7% of ASP funds since 2008. Publishers in this group do not receive funding every year: half (n=16) had received no funding in the most recent three fiscal years. Those who received at least some funding in the past three fiscal years (n=17) each received an average of $6,275/year (range $2,667 ‒ 13,333/year). All the specialist and generalist publishers and most French-language presses are in this third group.

ASP awards between 2008-17 are correlated with the number of applications submitted (Figure 5). In other words, funding in this period relates to extent of participation by the press in ASP.

Figure 5. ASP applications submitted and awards granted, total # by publisher 2008-2017

Data reported by Lobet & Larivière (2018) indicate that participation in ASP does not always reflect the size of output of the press, suggesting that some scholarly presses submit more of their manuscripts to ASP than others. Some possible reasons may include availability of other funding and press capacity. Specialty presses have received more funding from other federal funders than university presses in the past, and some have reported less capacity for the ASP application process^{Footnote 17}. University presses also range in size with corresponding capacity differences^{Footnote 18}. For example, in interviews, a small university press reported growing their participation in ASP once they had staff to ensure the quality of application packages and therefore likelihood of success.

In sum. ASP makes a very substantial contribution to a small subset of university presses. ASP’s processes are aligned closely with publishers, particularly in areas such as peer review where university presses have most capacity: this alignment has increased over time (see Section 3.2.2 below). Scholarly publishers of all types report ASP contributing to their ability to invest in scholarly books of limited cost-recovery potential, with secondary contributions in other areas (such as reputation).

Relationship of ASP to other federal funding

Two other federal agencies fund Canadian book publishers: Canadian Heritage (CH) and Canada Council for the Arts (CCA). High-level rationales for these agencies’ initiatives are similar to those expressed for ASP. They were described by CH and CCA representatives interviewed for the evaluation (n=2) and are also articulated in CH and CCA program documentation: Canadian publications are important but not always cost-effective due to small market (CH), and publishers are a mechanism through which organizational goals can be met because of their capacity to advance work in the field of focus for the agency (CCA).

CH and CCA’s criteria and funding mechanisms differ from ASP. Although publishers of ASP-funded books also access CCA and CH funding, ASP is the only federal funding that incentivizes publishers to take up SSH scholarly work.

Of note: while overlap is not indicated, there is also little synergy and points of tension among federal initiatives have been identified at the publisher level. As CH funding is revenue-based,^{Footnote 19} a few publisher respondents to the evaluation raised concern that OA publication might impact their CH funding. With respect to CCA, Lobet & Larivière (2018) report potential for reduced access to federal funding for specialist publishers under updated CCA guidelines (2017), as some nonfiction fits neither CCA nor ASP criteria, placing financial pressure on these publishers. This concern was also noted in four key informant interviews for the evaluation^{Footnote 20}.

Question 2. Do ASP objectives align with federal roles and priorities?

ASP objectives

Consensus was found in interviews with program representatives (n=4) and review of documents that ASP’s objectives align with federal roles and priorities. This is most evident with respect to priorities around building research excellence, in that ASP explicitly seeks to increase dissemination of Canadian SSH research and has an explicit focus on research excellence (see Appendix A). However, ASP’s objectives are very broadly stated, and this limits assessment of alignment at a more concrete level with SSHRC priorities or the objectives and goals of the parent Insight Program.

ASP in practice

“Though the ASPP views its program in terms of 'awards,' its longstanding relationship with Canadian publishers has been far more about making vital contributions to supporting the publishers in our mandates.” (University press representative)

As reported above, publishers prepare more than 90% of applications to ASP and receive and manage the funds in all cases. ASP is unusual for SSHRC in this respect. Although SSHRC does sometimes fund at the organizational level,^{Footnote 21} most SSHRC funding is applied for and received by researchers.

“Nobody publishes a book because someone gave us $8,000. We publish because we think the work is important.” (University press representative)

ASP pursues its objectives via the work of publishers, and for this reason the relevance of ASP rests largely on alignment between SSHRC and scholarly publisher mandates to support Canadian SSH research. Among publishers responding to the survey (n=14), 70% describe mandates that include activities aligned with SSHRC priorities, and priorities of the Insight Program under which ASP is located. Most mandate descriptions reference Canadian populations or regions. Examples include focus on la francophonie canadienne; Canadian economy, cultural and social topics (e.g., gender and race studies, labour, environment); and Indigenous studies. A few emphasize focus on scholarly work that is also of interest and accessible to non-specialist audiences, reflecting a knowledge mobilization mandate. A few reference broader mandates such as Canadian perspectives on global issues.

An estimated 44% of ASP-funded books between 2005-17 were focused on Canadian topics (Lobet & Larivière, 2018). Canadian authorship is part of eligibility requirements for ASP. Based on ASP data, half of all successful manuscripts between 2008-17 (50%) were outputs of SSHRC-funded research, and 32% of all successful manuscripts were led by first-time authors.

In the publisher survey, most respondents who described their mandate reported that the loss of ASP funding would change their ability to pursue that mandate (71%, n=10)^{Footnote 22}. These respondents all described the impact in terms of reductions in their scholarly publishing programs, with most impact on titles that have smaller readership (e.g., Canadian-focused or emerging research areas) or that require more editorial investment (e.g., first-time authors).

Equity, diversity and inclusion

ASP’s explicit objectives (Appendix A) do not currently reflect SSHRC’s EDI priorities. This is likely because ASP’s objectives predate SSHRC’s formalized EDI commitment. ASP collects monitoring data in two related priorities: program participation by first-time authors and French-language publications.

First-time authors. As noted above, about one-third of lead authors funded by ASP between 2008-17 were first-time authors (32%). First-time authors were generally as successful as non-first-time authors, with an average success rate of 68.3% vs 71.2%. Submissions from first-time authors have increased over time, especially after 2013.

Francophone authors. ASP data show lower than expected submissions and success rates for French-language manuscripts. This likely relates to systemic barriers for francophone authors.

Submissions. Between 2008-17, most manuscripts submitted to ASP were in English (85%), with 15% submitted in French. The proportion of French submissions is lower than expected, given that 23% of respondents to the researcher survey report French as their primary language in the workplace,^{Footnote 23} and an estimated 22% of Canadians report French as their mother tongue.^{Footnote 24} Of note, French-language submissions to ASP have varied over the study period from a high of 21% in 2008 to a low of 8% in 2013, rising again to nearly 18% of submissions by 2017.^{Footnote 25}

Although 23% of researcher survey respondents report French as their primary language at work, when asked which language they typically use for book and journal article manuscripts, only 12% of respondents report French, 80% report English and 8% responded “it depends.” Respondents were asked if they had experienced any language barriers with respect to publishing their research. Of those responding in French, 61% (n=79) reported experiencing at least some barriers, compared with 9% of those responding in English (n=235).^{Footnote 26} When asked to describe barriers, francophone researchers referred to lower value placed on French publications affecting career advancement and creating pressure to write in English, i.e., related to research assessment practices. A variety of more specific examples were provided by individual respondents, including reduced productivity due to writing in a second language and being asked to not cite French publications in the reference lists of English-language publications.

Success rates. French-language manuscripts have had a lower overall success rate with ASP than English-language manuscripts. Among manuscripts for which a final decision is recorded in 2008-17, 65.6% of French-language manuscripts were approved for funding vs 75.3% of English-language manuscripts.

This appears to relate to applications submitted by authors vs applications submitted by publishers. Applications submitted by authors have a lower overall success rate than those submitted by publishers, regardless of language (45.6% vs 75.2%), and French-language manuscripts are submitted more often by authors than are English manuscripts (21% vs 2%). When success rates by language and source of application are considered together, the gap between French and English-language is smaller, as shown in Table 1.

The higher success rate for manuscripts submitted by publishers likely relates to contributions publishers make to the manuscript before submission to ASP (see below, performance and quality). Although not studied in depth by this evaluation, the reason for a higher proportion of author-submitted applications among francophone authors may reflect structural differences in the publishing system,^{Footnote 27} differences in access to university presses and/or participation by francophone university presses in ASP.

Publisher EDI activities. As noted above, the degree of alignment between publisher and SSHRC priorities is important in the context of ASP. Publishers were asked in the survey if their press had undertaken any activities explicitly aimed at increasing equitable and inclusive participation in scholarly publishing (for example, efforts to increase the diversity of the editorial or publications board or to increase diversity of topics or scholars published). Just over half (55%) replied “yes.” They were also asked if any such activities are planned for the next 12 months: just under half (45%) replied “yes.”

Description of EDI activities ranged from general statements of commitment to equity and reference to individual book projects, to specifically serving minority communities either as a focus of the press or as a dedicated program. For example, references were made to francophone and Indigenous authors.

In sum. ASP’s objectives align generally with federal priorities in terms of research dissemination. ASP’s objectives are very broadly articulated, limiting assessment on a concrete level with SSHRC or Insight Program priorities. The funding mechanism is not designed to allow actively targeting funds to areas in the research community where relevance of ASP appears highest, or where opportunities are indicated to advance inclusion or equity, such as for francophone authors.^{Footnote 28} However, in practice, the data suggest alignment with some priorities, e.g., in the proportion of funded manuscripts on Canadian topics or by first-time book authors. The funding is reported to increase capacity of publishers to pursue mandates to support Canadian research and researchers.

3.0 Performance

Question 3. What contribution has ASP funding made to quantity, quality and dissemination of published Canadian SSH research?

The three parts to this question, quantity, quality and dissemination, are reported separately below. The dissemination component focused on accessibility and discoverability because this was a joint evaluation with ASJ and ASJ actively incentivizes activities in this area. This component was studied in less depth for ASP, but some information is provided below as background for the program.

3.1 Conclusions

Quantity. The findings indicate a contribution to quantity. This contribution is largely indirect, through an increase in publisher capacity to invest in scholarly books.

Quality. ASP does not contribute directly to the quality of individual manuscripts. However, the competitive nature of ASP may have an indirect effect by incentivizing additional work at the publisher level to raise review ratings before applying to ASP. ASP requires adherence to guidelines for peer review, which may support some consistency across applications.

Dissemination. ASP contributes to dissemination of research in simple terms by increasing the number of publications. There is some limited evidence of a small, secondary contribution of ASP in other areas, for example increased publisher capacity for marketing and promotion.

3.2 Detailed findings

3.2.1 Contribution to the quantity of research published

ASP was proposed to contribute to the quantity of published research as follows:^{Footnote 29}

• Direct contribution. The ASP award for each book provides a direct input to the publication cost of that book, without which the book would not be published.
• Indirect contribution. Publishers make decisions about size of their overall publishing program based on anticipated funding from ASP.

Direct contribution

Past evaluations^{Footnote 30} have assessed whether ASP makes a direct contribution to the number of published books by asking successful applicants to consider what would have happened with their book project had it not been awarded funding, and asking unsuccessful applicants about outcomes of their unfunded book projects^{Footnote 31}. Overall, these evaluations concluded that a majority of books denied ASP funding would have been published anyway. A similar finding was reported by Lobet & Larivière (2018).

These previous findings are also supported by data from key informant interviews with publishers for this evaluation. The majority of publishers interviewed (81%, n=9) reported they would attempt to publish a book, even if unsuccessful with ASP, because of the investment already made in the book. Key informants also reported, however, that availability to authors of alternate funding is an important factor in their ability to publish without ASP. This suggests that lack of ASP funding may be differentially felt by some researchers because resources available to authors may relate to their discipline, career stage and/or institutional affiliation. Book publications also typically reflect several years of work for a researcher and do not fall easily within the funding period of SSHRC grants to researchers.

The outcomes of 120 unsuccessful applications from 2013 were traced for this evaluation to further assess the possibility of a direct effect of funding: 13 of these 120 manuscripts were resubmitted to ASP at a later date and approved. The remaining 107 did not receive ASP funding: 90% of these were later published (n=96); 83% were published within two years of being denied funding (2013-15); and 79% were published by the same publisher named on the application.

The number of books published after delay of more than two years (n=7) or with a different publisher (n=9) was too small for subgroup analysis to investigate differential impact, for example by discipline or career stage of the author. Overall, however, 21% of the cohort (n=23) were delayed, had to change publisher, or could not be found and are assumed to have been not published.

Publishers were advised in the survey that other data had indicated that a majority of unsuccessful books are published without ASP funding. Publishers were asked if this was accurate for their own press: 58% agreed (n=11). However, one-third responded in the negative (32%, n=6).

Publishers who agreed that this was accurate for their press (n=11) were asked how they typically compensate for the absence of ASP funding. Multiple approaches are taken, primarily drawing from the existing budget of the press (83%) and requesting the author find other funding (83%, see Figure 6).

Figure 6. Methods to compensate for lack of ASP funding to enable publication of a manuscript (% of publishers identifying each method) (n=11)

Publishers were asked if unsuccessful books were affected in any other way by a lack of ASP funding. All but one respondent replied “yes” (n=17). Fifteen publishers provided information on other types of impact, including delays to the project while other funding is sought, reductions in production costs (e.g., fewer images) and reductions in marketing/promotion, even if other funding is found as the amount may not be comparable to ASP. If other funds are not found, some publishers reported that they would then abandon the project.

In sum. While some manuscripts would be abandoned if not funded by ASP, the majority are published anyway by the applicant publisher and others are published elsewhere (by other publishers).

Indirect contribution to quantity

Data suggest ASP funding acts to increase the number of scholarly books published in Canada. This is primarily by influencing publisher decisions about future books based on their history of funding from ASP and expectation of future funding.

“ASPP funding not only keeps our program going but it allows us to support and champion emerging disciplines, emerging academics’ new works, and critical works that are not yet mainstream…The ASPP not only underwrites the ability to take risks … but also is a backbone to how the press finances itself. It underwrites basically the premise of a university press in Canada, I think.” (University press representative)

Figure 7. Total new book projects projected by publishers with and without ASP (n=11)

Figure 7 long description

Figure 7. Total new book projects projected by publishers with and without ASP (n=11)

This bar chart shows the number of scholarly publication projects planned for the next 12 months versus what would be planned without ASP funding. Projected new projects with ASP funding = 721. Projected without ASP funding = 406.

Publishers were asked in the survey to estimate the number of new scholarly book projects their press will accept for review over the next 12 months and, as a follow up question, how many they would be likely to accept if there was no opportunity to receive ASP awards. A total of 11 publishers were able to provide estimates for both scenarios.^{Footnote 32} Individual responses varied, from no change with or without ASP (n=3) to no scholarly books produced at all in the absence of ASP (n=1). Remaining responses ranged between a 25% to 61% estimated reduction in scholarly book projects, depending on the publisher. Collectively, these 11 publishers report plans to accept 721 new projects for review in the next 12 months but estimate a total of 406 if ASP were not available, an overall estimated reduction of 45% (Figure 7).

Data reported above in the section on relevance support these results. Participants in key informant interviews and respondents to the survey to publishers consistently and universally reported that ASP contributes to the overall amount of scholarly research they can publish: 68% of survey respondents (n=19) reported that an absence of ASP funding would have a major adverse impact on the number of scholarly books they produce. Data reported by Lobet & Larivière (2018) on 11 university presses over the period 2008-16 suggest that ASP contributed funds to 28% of titles published by these presses over that period on average, and to 50% of titles of the three presses participating most in ASP (Group 1, Figure 5, above).

Differential impact by type of book. In the survey, publishers were also asked whether a lack of ASPP funding would impact decisions about the type of book they accept. These responses were consistent with Phase 1 data and responses to other questions in the survey, namely that the impact would be felt most by those books with the least likelihood of cost recovery, including: Canadian topics, Canadian authors, first-time authors, and topics with specialized audiences or in new fields.

“An absence of ASPP funding would mean for my press that we would need to publish fewer Canadian monographs, or monographs [ad]dressing Canadian subjects for which we feel the market is small or uncertain. This will disproportionately and negatively affect emerging scholars and fields of study, diverse populations, and work with a regional focus. It will place a particular challenge to supporting Indigenous scholarship, for example.” (University press representative)

“[B]ooks by first-time authors most often require a greater investment of editorial and production staff resources. Without ASPP funding, we would have to limit, quite severely, the number of first-time authors we publish. Similarly, certain fields sell better than others, sometimes substantially, so again we would almost inevitably have to choose to publish scholars working in more ‘popular’ fields.” (University press representative)

In sum. The data suggest that ASP does increase the number of scholarly books published, primarily through increasing publishers’ anticipated capacity for risk on titles of high scholarly value and/or aligned with their mandate but not commercially viable. Although, at an individual book level, most books that are not successful with ASP are published anyway, the absence of ASP funding entirely would reduce Canadian scholarly publishing capacity. The proposed disproportionate effect on some researchers and topics may affect those who are reported to face challenges publishing outside of Canada. Note that this conclusion is based mainly on self-report data by recipients of funding and, given the absence of baseline or comparative data, the findings are not definitive. However, data are consistent across multiple sources and methods in support of this conclusion.

3.2.2 Quality outcomes

The evaluation did not find evidence that ASP contributes directly to the quality of manuscripts it funds. ASP does not have a mechanism in its design to contribute to the quality of individual manuscripts directly, and most manuscripts are already highly developed by publishers before submission to ASP.

All publishers interviewed for this evaluation (n=11), including both specialty and university presses, reported that ASP does not play a role in the quality of manuscripts because the quality is based within the research and on quality standards that the presses are motivated to uphold regardless of ASP, including peer-review standards.

Since 2014, publishers have had the option to provide the two peer reviews required for a manuscript with their applications, and respondents to the survey confirmed that they always submit both peer reviews to ASP.^{Footnote 33} Alternatively, ASP will arrange for one of the two peer reviews if both are not supplied. Two respondents stated that control over the peer-review process allows for important editorial work to take place, as well as recruitment of appropriate reviewers who will contribute to the quality of the manuscript. In other comments, most respondents replied that they would only submit if the manuscript had received excellent reviews because the chances of success with ASP are otherwise low. To a separate question, 79% of publisher respondents to the survey reported the ratings of peer reviewers were important or very important to the decision to apply to ASP.

Publishers were asked in the survey how long their press typically works on a book project before submitting an application to ASP. The mean response was 7.6 months and the median was 6.0 months.

“There is often a long way between even the best submitted manuscripts and what in our view is publishable. Where we see potential in a manuscript, we give direction to authors, both before and after peer review, in revising their manuscripts. Often, we will engage a re-review by one or more of our original peer reviewers... This process, we have found, improves the odds of a manuscript winning the ASPP lottery. Whether or not this ultimately results in a book that we publish, we see this endeavour as part of the service that we, as a university press, can contribute to the scholarly research process.” (University press representative)

When both reviews are submitted by the publisher, ASP does not review the entire manuscript. The ASP review committee considers the submitted peer reviews, the introductory and concluding chapters of the manuscript, table of contents and bibliography. Although authors are able to submit to ASP directly, and ASP will coordinate both peer review of these manuscripts, applications made by authors have an average success rate of more than 25% lower than manuscripts submitted by publishers (see relevance section, above). This is a primary reason why authors are encouraged by ASP to submit their manuscripts through a publisher rather than directly to ASP^{Footnote 34}.

ASP funding may contribute to quality indirectly by increasing publisher capacity to maintain their quality processes. When asked to project impact of loss of ASP funding, 58% of publisher respondents (n=11) reported at least some impact on their ability to apply quality assurance processes and the extent of support they provide to authors, although one-third (n=6) reported no impact in either of these areas.

ASP requires a standard of single-blind peer review and adherence to conflict-of-interest guidelines, and in this way provides a form of audit function. ASP also provides a peer review form that publishers are required to use, which may provide some consistency in coverage of reviews across applications. In the survey and interviews, most respondents described their press routinely following a more extensive peer-review processes than required by ASP, while a few described the ASP’s requirements as excessive.

3.2.3 Dissemination outcomes

The strongest evidence of a contribution by ASP to dissemination of research is simply by contributing to the amount of research available in published form. This is the contribution ASP has been designed to make and ASP does not provide incentives in specific areas of accessibility or discoverability. There is limited evidence of secondary contribution of ASP in other areas. This is reported below with other information collected to provide information to ASP.

Marketing and promotion capacity. 45% of publishers responding to the survey (n=9) report that ASP has been important or very important in their ability to pursue marketing and promotion of scholarly books. This is also an area where about half of publisher respondents (53%) projected major impact in the absence of ASP. Descriptions of marketing and promotional activities include promotion at events, advertising, submissions of books for prizes and book reviews, catalogue listings and social media promotion. A few respondents to the survey highlighted the importance of distribution networks to enabling discovery by readers, i.e., partnerships between aggregators, distributors, wholesalers and the publishers’ own sales personnel. One respondent remarked that current practices to increase discoverability are a reflection of how academics usually seek new books in their field.

Translation. ASP offers translation grants to increase access to research across languages, but only five grants are offered each year. These grants have experienced lower than expected uptake, which has been attributed to the small amount of the grant relative to cost of translation and publication (Lobet & Larivière, 2018). Of publishers responding to the survey, 84% (n=16) report being aware of the grant and half (53%) have applied for a translation grant.

Digital publication. In the survey, publishers report that most books are published in both digital and print (76%), with a slightly higher percentage of ASP-funded books published in both formats (81%). A minority of books continue to be published print-only (23% overall and 19% of ASP-funded). Publisher participants in key informant interviews (n=11) did not attribute their transition to digital publishing to ASP. This is supported by data from document reviews suggesting that the size of ASP is insufficient to change practices in the sector at this level. Interview participants reported a continued strong preference for print books among audiences and a weak market for digital books. As a result, publishers who have transitioned to digital continue to maintain print publishing.

Discoverability. Publishers were asked to indicate if, over the past 12 months, their press had used any of a set of discoverability practices compiled from Phase 1 results, a framework developed by the Scholarly Publishing and Academic Resources Coalition (SPARC) and Public Library of Science (PLOS)^{Footnote 35} and consultation with publisher representatives piloting the survey. Most publishers indicate they have undertaken indexing of published books (85%) and have allowed book metadata to be machine readable (65%). One-quarter (25%) of respondents were unsure if they had used MARC records, standard document identifiers or international standard author identifiers, although one publisher noted that some of these practices are undertaken by book distributors and wholesalers. One respondent noted use of other metadata standards specific to the book industry (e.g., ONIX fields).

Publishers were also asked how important ASP funding has been to their ability to pursue each of the discoverability practices they report: 44% of those indexing published books said that ASP funding has been very important or important to their ability to pursue the practice (n=7), but 37% reported ASP to be of minimal or no importance. 40% of those who indicated that their press has undertaken other activities (e.g. machine-readable metadata) reported that ASP funding has been very important or important in this (n=6), but 47% reported that ASP was of minimal or no importance.

Open Access. Publishers were also asked in the survey to report the share of their scholarly books overall and the share of their ASP-funded books they currently make available in OA. Among those providing an answer, 14.6% of all scholarly books published in the past 12 months were made available as OA (16 respondents) and 16.6% of ASP-funded books were made available as OA (13 respondents). Publishers reported that 10% of their OA titles are backlist titles (i.e., previously published in non-OA, and later made OA). Of the 20 publisher survey respondents, three report plans to increase OA publication over the next two years; 10 report no plan to increase OA publication; and six are unsure.

OA enablers and barriers. Publishers identified enablers and barriers for their press related to OA publication. All respondents to this question (n=8) identified the importance of funding. In addition, two university press respondents mentioned philosophical support from their academic institution, and one respondent noted growing awareness and perceived credibility of OA, and improvements in use of metadata, social media and other promotion to address structural barriers to OA in the distribution system.

Publishers were asked to indicate what sources of revenue and/or funding support have helped offset the cost of publishing OA. The most frequent responses were external funding to the press from government or foundations (63%), support to the press from an academic institution (50%) and funds raised by authors (50%, see Figure 8).

Figure 8. Publisher reported sources of revenue and/or funding to offset cost of OA (n=8).

“As it stands, we aim to break even through sales over the first three years of the life of a title. We do not always succeed… even a mission-driven publisher with a cost-recovery business model cannot make books O/A as a matter of course, as it would undermine our ability to sustain our publishing program.” (University press representative)

The same respondents (n=8) also reported barriers to OA publication. Nearly all respondents to this question reported lack of funding and/or impact on revenue as the most important barrier. One respondent noted that they could not publish a book in OA that had market potential, as the revenue from sales of these books subsidizes other aspects of their publishing process. Three respondents also mentioned potential impact on other (revenue-based) funding as a barrier; two of these named the Canada Book Fund.

Distribution system. A second barrier mentioned by multiple respondents was that the distribution system disfavours OA books, because most aggregators do not carry OA books and library wholesalers do not make OA books visible. As reported by one respondent: “Discoverability has been a challenge, as the library wholesalers from whom most institutions receive information about publications do not, understandably, include the free option [of the book] in the data they transmit to their customers. Improved metadata, social media marketing and other promotion have helped us offset this” (University press representative).

Other reported barriers include author disinterest and intellectual property rights, and restrictions in some communities about circulation of knowledge and cultural material (e.g., among Indigenous communities). With respect to author interest in OA, respondents to the researcher survey who had published with a Canadian book publisher (n=164) were asked about the importance of OA options for selection of a book publisher. Nearly half (46%) replied that OA options are of little or no importance to them (Figure 9). The results were similar across language, disciplinary categories and career stage^{Footnote 36}.

Figure 9. Importance of OA options in selection of book publisher, authors of Canadian-published titles (n=164).

These results suggest less interest in OA book publication compared with OA journal publication. A similar question was asked about importance of OA for selecting a scholarly journal publisher to all researchers in the survey (n=314). To this question, 19% replied that OA options were important or essential and 25% replied OA options were of little or no importance at all. The difference may in part be due to OA having become more established in journal publication than in book publication to date.

In sum. While ASP is not designed to incentivize activities related to accessibility and discoverability of books, some secondary contribution is suggested by publishers, primarily with respect to increased capacity for marketing and promotion. Scholarly book publishers report low capacity for the risk involved to transition to OA or ability to replace revenues that would be lost through OA. Structural barriers to OA and weak demand in the research community are also reported.

4.0 Cost efficiency

Question 4. Is ASP delivered in a cost-efficient manner?

4.1 Conclusion

ASP’s cost efficiency ratio is substantially higher than the average for Research Grants and Partnerships. This is attributable to two characteristics of ASP:

1. The funding mechanism, which requires substantial resource investment per application.
2. Minimal opportunity for economy of scale. ASP has a more resource-intensive funding model but only half the grant funds of ASJ (for example). Moreover, unlike ASJ, ASP is unable to share overhead costs with other programs delivered by SSHRC (e.g., IT, accounting).

In other words, ASP’s cost efficiency ratio is high, but for structural reasons rather than inefficiencies in its delivery. Improving ASP cost efficiency in a substantial way would require changes to the funding delivery mechanism.

4.2 Detailed findings

SSHRC typically assesses cost efficiency by calculating the ratio between a program’s operating costs and grant funds awarded (the operating ratio). Data for this analysis were supplied by the office of Awards Management, Compliance and Accountability at SSHRC. As ASP is delivered by the FHSS, additional data were collected from FHSS public reports on grant funds allocated, in order to isolate operating expenditures from award funds disbursed. The quantitative results were also reviewed with two representatives of FHSS management in a one-hour group interview to contextualize the results and gather information on primary cost drivers for ASP.

ASP efficiency ratios are presented in Table 2 for the four fiscal years 2015-16 to 2018-19.

Over this period, ASP cost between 19.5ȼ to 25.0ȼ to administer for every $1.00 granted, with an average of 22.4ȼ/$1.00 over the period. Overall operating expenditures for ASP ranged from 16.3% to 20.0% of total program expenditures, with an average of 18.3% over the period. The operating ratio for ASP has remained fairly consistent over the period. Notably, the evaluation of ASP in 2004 reported administrative costs of approximately 22% of the budget (Goss Gilroy Inc., 2004), indicating that administrative costs have fallen slightly over the past 15 years, despite inflation.

SSHRC does not have other opportunities that operate on a similar model. As a result, we were unable to directly compare cost efficiencies with another program. The average cost efficiency for programs in the Research Grants and Partnerships portfolio, in which ASP is included, was 3.84ȼ/$1.00 in 2018-19. Representatives from FHSS were asked about the primary costs to deliver ASP. Three primary cost categories were identified:

1. Human resources. HR costs include the salary of a full-time, bilingual program officer, who is responsible for overall program coordination, including coordinating the review of manuscripts. In addition, four other FHSS staff allocate a portion of their time to ASP. There are also episodic HR costs typical to an organization, for example, training and onboarding of new staff.
2. Promotion. This category includes raising awareness of ASP among researchers. Examples include communications via the FHSS website, events for emerging scholars at the Congress of the Humanities and Social Sciences, and participation in French-language publication events to increase program awareness among francophone academics and publishers.
3. Infrastructure / capital. A database for ASP carries some cost under this category.

5.0 Alternatives

Question 5. What viable alternative approaches have been identified that SSHRC could consider to increase dissemination of original Canadian research results in the social sciences and humanities?

In consultation with the EAC, this question was directed at describing government interventions to support OA publishing as reported in the peer-reviewed literature. This literature focuses almost entirely on journal publication, and findings from the review are therefore included in the report for ASJ. A very brief synopsis is provided below. For ASP, data were also collected on strengths and weaknesses of the current funding delivery mechanism, given changes in the context and the amount of time since the last evaluation.

Strengths and weaknesses of the ASP model. ASP operates on a model of continuous intake of applications and individual small grant (per book) adjudication. Applications for about 250 manuscripts are processed each year and funds to awarded books are dispersed after each book is published, on receipt of publication information (Appendix A). Interview participants (including publishers, sector stakeholders and program management) and publisher survey respondents volunteered comments on strengths and weaknesses of the ASP funding model, as follows:

Strengths:

• As a merit-based funding model, ASP avoids incentivizing quantity over quality, and provides recognition to authors and publishers.
• ASP brings authors and publishers together in a pragmatic way to contribute to the development of each book. Manuscripts are well developed before funding is committed.
• A strong sense of ownership over the program is felt by the scholarly and publishing community, attributed in part to participation as reviewers, authors and editors of books.
• Authors can direct the award to a publisher of their choice, so long as the publisher is ASP-eligible^{Footnote 37}, although most applications are submitted by publishers and redirection is rare.
• Publishers are able to use the funds flexibly, which is important given diversity of books, readers and publishers.
• ASP is not revenue-based and therefore does not disincentivize OA publication.
• In the survey, most publishers (79%, n=13) agreed “to a great extent” that ASP’s model creates additional incentive for publishers to publish books of high scholarly value, even if not commercially viable. A majority also reported that, in their view, the model “to a great extent” ensures that public funds directly support scholarly books (77%) and ensures that funds go to books that make a substantial contribution to the relevant field or discipline (71%).
• About half of publisher respondents to the survey (53%) agreed “to a great extent” that the process was fair for different types of authors.

Weaknesses:

• ASP is resource-intensive for SSHRC and FHSS. Operating cost to funds awarded is very high.
• The individual adjudication process delays the publication of both successful and unsuccessful manuscripts.
• Applications are resource-intensive for publishers, requiring capacity to participate in ASP which may advantage some publishers.
• Ambiguity around whether ASP is an author or a publisher program creates challenges for stakeholders to understand ASP and for program planning.
• ASP’s objectives are broad and ambitious, but the funding envelope is small.
• There is minimal flexibility to target funding to priority areas or areas of relevance in the research community.
• In the survey, publishers provided mixed responses on whether the process is fair for different types of publishers (40% agreed to a great extent; 40% slightly agreed; 20% did not agree).
• Most publisher survey respondents did not agree or agreed only slightly that ASP provides important feedback to authors (35% did not agree; 41% agreed slightly) or increases peer-review quality (47% did not agree; 42% agreed slightly).

In sum. Although ASP’s funding model is resource-intensive for multiple parties involved, it has some advantages to be considered if alternative delivery mechanisms are explored.

Interventions by governments for OA publishing. Four primary categories of government funding interventions to increase OA were identified in the literature:^{Footnote 38} funding directly to researchers (for example, to pay OA processing charges); institutional-level funding (e.g., block grants to academic institutions to pay OA charges on behalf of affiliated researchers); publisher or journal-level funding (e.g., to provide OA publication options at low or no charge to authors); and system-level funding for OA infrastructure (e.g., OA portals, workflow software, funding consortia).

Advantages and disadvantages have been documented for all the approaches identified in the literature. Primary themes in this work include the strong influence of researcher incentive systems on OA uptake, including practices for assessment of research publications in academic institutions. Attention to incentive systems and research assessment methods^{Footnote 39} are reported to be important for any strategy to support transition to OA. A second primary theme relates to unanticipated effects of funder interventions, which highlight the limited direct control of funders over outcomes and the dynamic nature of the environment. This suggests consultation and ongoing monitoring are necessary to mitigate adverse effects. A final theme is the importance of alignment among various funder policies and a need to support the development of common standards and best practices for discoverability in the sector.

6.0 Conclusions and recommendations

6.1 Conclusions

Relevance

ASP is relevant to researchers; however, importance is highly concentrated in specific SSH disciplines and areas of research. While researchers participate in ASP from a wide variety of SSH disciplines, ASP is used most intensively by those in a subset, both in humanities (e.g., history, languages and literature) and social sciences (e.g., political science). Book-length format is rated by a majority of surveyed researchers in these disciplines as highly important for publication, source material and career progression. ASP is also relevant to researchers who report particular added value from collaboration with Canadian publishers (e.g., first time authors, authors writing in French, researchers working on topics that require Canadian contextual knowledge). Overall, an estimated 16.7% of Canadian SSH researchers have published with a Canadian book publisher. Among those surveyed in this group, about one-third described adverse impact on their book if they were not able to publish in Canada.

ASP is highly relevant to Canadian scholarly publishers, especially university presses. ASP contributes to publisher capacity to invest in books that are of high scholarly value but have limited potential for cost recovery (i.e., due to a specialized or national audience). There is no equivalent to ASP funding among other federal programs, and other federal funding does not incentivize scholarly publication. In the period under study, over 90% of ASP awards went to a subset of 10 university presses.

Alignment is evident between ASP and SSHRC priorities. ASP’s objectives are very broadly articulated, limiting assessment of alignment at a concrete level with SSHRC or Insight Program priorities. The funding mechanism is not designed to actively target funds to specific priority areas or to areas in the research community where relevance of ASP appears highest. However, the data suggest alignment in practice with some priorities, e.g., an estimated 50% of awards are to outputs of SSHRC-funded research, 44% to manuscripts on Canadian topics and 32% to those led by first-time book authors.

In practice, the relevance of ASP rests on the alignment between SSHRC and scholarly publisher mandates to support Canadian SSH research. This is because ASP pursues its objectives via the work of publishers. Although ASP is explicitly a program for researchers (as an award for authors), more than 90% of applications are developed and submitted by publishers: the publisher selects which manuscripts to submit, coordinates peer review, invests in editorial development of the manuscript before application, submits the application, and receives and manages the awarded funds.

ASP’s objectives do not yet incorporate EDI. ASP collects monitoring data in two related areas: participation by first-time authors and French-language publications. The data indicate ASP is important for first-time authors. Challenges exist related to systemic barriers for francophone authors.

Performance

The findings indicate that ASP is achieving its primary intended objective to support research dissemination by contributing to the amount of SSH research published. Given the absence of baseline or comparative data, the findings are not definitive. However, data across multiple sources support the conclusion that ASP does increase publisher capacity to take risk on titles of high scholarly value and/or aligned with their mandate but not commercially viable. The absence of ASP would reduce Canadian scholarly publishing capacity, with disproportionate effect on some researchers and topics.

The evaluation did not find evidence that ASP contributes directly to the quality of manuscripts it funds. ASP does not have a mechanism to contribute to the quality of manuscripts directly, although it upholds a consistent peer-review standard for submissions. As quality processes are already embedded in upstream research and scholarly publication practices, ASP may be best positioned to contribute through indirect support, i.e., by directing funding to where quality standards are demonstrably high.

ASP is not designed to incentivize activities related to accessibility and discoverability of books. However, a small secondary contribution of ASP is reported by publishers, primarily with respect to increased capacity for marketing and promotion. Scholarly book publishers report low capacity for the risk involved to transition to OA or ability to replace revenues that would be lost through OA. Structural barriers to OA and weak demand in the research community are also reported.

Cost efficiency

ASP’s cost efficiency ratio is high with operating costs at 22¢ per $1 in grant funds awarded. This ratio is substantially higher than the average for Research Grants and Partnerships. This is attributable to two characteristics of ASP:

• The funding mechanism, which requires substantial resource investment per applicant.
• Minimal opportunity for economy of scale. ASP has a more resource-intensive funding model but only half the grant funds of ASJ (for example). Moreover, unlike ASJ, ASP is unable to share overhead costs with other programs delivered by SSHRC (e.g., IT, accounting).

In other words, ASP’s cost efficiency ratio is high, but for structural reasons rather than inefficiencies in its delivery. Improving ASP cost efficiency in a substantial way would require changes to the funding delivery mechanism.

Alternatives

Both strengths and weaknesses were reported for ASP’s funding model. As examples: the current model avoids incentivizing quantity over quality and offers flexibility in use of funds, which is important given the diversity in SSH books and in the publishing sector. However, the process delays publication of manuscripts and provides minimal flexibility for SSHRC to target funding to priority areas or where relevance is most concentrated. Other strengths and weaknesses are documented above. Although ASP’s funding model is resource-intensive for the multiple parties involved, including authors and publishers, it has advantages to be considered if alternative delivery mechanisms are explored.

6.2 Recommendations

Recommendation 1. Continue to offer support for long-form publishing of Canadian SSH research. The funding fills a niche not addressed by other funding, is relevant to Canadian researchers, and provides capacity for the publication of research that is important for SSH and Canada.

Recommendation 2. Develop more fully articulated and concrete objectives for ASP. ASP’s objectives are broad and ambitious for its small size and the concentrated relevance of the funding for SSH research. A set of more concrete objectives is needed for program management and SSHRC to situate and guide the funding opportunity.

The following should be considered:

• ASP is currently a funding opportunity within SSHRC’s Insight Program. Objectives should consider the goals of ASP’s parent program and where the funding is most needed to advance SSH research.
• A debate about whether ASP is an author program or a publisher program has existed for at least 30 years^{Footnote 40}. The ambiguity is confusing for stakeholders, complicates program development and is essentially a false dichotomy. ASP can be understood as an initiative that serves both authors and publishers working for a common purpose to advance SSH research and the benefits of that research to society. In other words, ASP supports Canadian SSH research and researchers, and it does so by providing capacity to the work of scholarly publishers who serve the same community. It would be helpful to recognize this explicitly in ASP’s description and logic model, taking care to separate the ends (advancement of Canadian SSH research) from the means (collaboration with Canadian scholarly publishers).
• ASP’s objectives do not currently incorporate EDI. At minimum, taking steps within ASP’s capacity to collect monitoring data related to EDI should be considered.

Recommendation 3. Identify options to update ASP’s funding mechanism. SSHRC is currently working collaboratively with FHSS and the Association of Canadian University Presses (ACUP) to develop OA capacity in scholarly book publishing and to increase French-language participation in ASP. In concert with those efforts, SSHRC should explore alternate delivery mechanisms for ASP to improve cost efficiency and performance. Opportunities would need to be weighed with consideration for the key advantages offered under the current model.

• In terms of cost efficiency, the individual (per book) adjudication model should be reconsidered. Moving away from the individual book adjudication process would not likely reduce the overall quality of publications. The current funding model does not appear to contribute to the quality of manuscripts other than through guidelines for a consistent peer review. Scholarly publishers have strong incentive to maintain the high quality and scholarly standard of their publications, especially given the importance of reputation to researchers when selecting a publisher. The findings suggest ASP increases publisher capacity to invest in manuscripts that these publishers would be motivated to support regardless of the delivery mechanism used for ASP.
• In terms of performance, greater flexibility is needed for SSHRC to target its investment to areas where ASP funding has highest relevance and potential impact, given the small size of the funding envelope. The current model does not provide the flexibility to be responsive to a more concrete set of objectives (see above).
• As options are explored, consideration should be made to SSHRC’s commitments to official language minority communities.^{Footnote 41} As reported above, representation of French-language publications in ASP may be influenced in part by factors outside ASP’s purview, for example in current practices for research assessment in institutions. Multifaceted action by SSHRC may be needed.^{Footnote 42} However, increasing the flexibility of ASP’s funding mechanism may also be helpful to enable response to priorities like this.

Alternative models proposed in past reports^{Footnote 43} have included converting ASP to a “block grant” for scholarly book publishers or to academic institutions or adding to existing funding for researchers who publish in book form. Other possibilities offered by participants in the current evaluation include adopting a periodic competition model similar to ASJ, which could allow for efficiencies as well as criteria or incentives in line with SSHRC priorities.

While exploring options to update ASP, attention is needed to opportunities in the sector for collaboration. ASP is not large enough to effect change on its own. In particular:

• ASP currently leverages the expertise and capacity of scholarly publishers whose activities align with SSHRC’s mandate to support and promote SSH research. Benefits of collaboration with Canadian publishers are reported above, particularly but not exclusively for some groups of researchers (e.g., first-time authors, researchers working on topics of national interest or emerging topics and French-language researchers).
• Improving synergy with other federal funders is recommended. Any tensions with other active initiatives will limit ASP’s effectiveness. Improving synergy with Canadian Heritage initiatives^{Footnote 44}, which include the Canada Book Fund and Support for Organizations, is particularly important in the context of SSHRC’s priority to advance OA.

Recommendation 4. Update ASP’s program logic model or theory of change model. A new program logic model or theory of change should be drafted for ASP. The findings reported above provide the basis for an empirically based change model and the Evaluation Division can provide technical assistance. This form of documentation would be useful for exploring alternatives going forward.

References

• Bamberger, M. (2012). Introduction to mixed methods in impact evaluation.
• Best, A., & Holmes, B. (2010). Systems thinking, knowledge and action: towards better models and methods. Evidence & Policy: A Journal of Research, Debate and Practice, 6(2), 145-159.
• Canadian Heritage (CH). (2019). Evaluation of the Canada Book Fund 2012-13 to 2017-18.
• Centers for Disease Control (CDC). (2013). Mixed methods in program evaluation.
• Chen, H. (2006). A theory-driven evaluation perspective on mixed-methods research. Research in the Schools, 13(1), p. 75-83.
• Fast, E. (2018). Open access and the ASPP: report to the Federation of Humanities and Social Sciences. Seja Strategies.
• Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank Quarterly, 82(4), 581-629.
• Lobet, D., & Larivière, V. (2018). Study on the Awards to Scholarly Publications Program and the Scholarly Book Situation in Canada. Federation for the Humanities and Social Sciences.
• Patton, Michael Quinn.(2008) Utilization-Focused Evaluation: 4th edition. Thousand Oaks, CA: Sage Publishing.
• Ryan, K., Gannon-Slater, N., & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small-and medium-scale evaluations. American Journal of Evaluation, 33(3), 414-430. *need doi
• SSHRC (2018). Definitions of Terms.
• Tashakkori, A., & Teddlie, C. (2010). The SAGE handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage Publishing.

Appendix A. Profile of ASP

Overview

ASP is a competitive funding opportunity designed to assist with the publication of SSH scholarly monographs, books and other long-form publications. ASP has supported the publication of over 7,000 books over its lifetime. ASP is funded by SSHRC and administered by the Federation for the Humanities and Social Sciences (FHSS).

Objectives

ASP’s overall objective is to enrich the social, cultural and intellectual life of Canada and the world.

ASP’s intermediate objective is to support research dissemination and encourage excellence in humanities and social science scholarship by supporting books of advanced scholarship in the humanities and social sciences that make an important contribution to knowledge.

History

ASP has existed since 1941. Originally called the Aid to Publication Program, ASP was first designed to offset the cost of publishing book-length scholarly manuscripts in Canada and to increase opportunity for Canadian scholars to publish their work given the small size of the Canadian market for scholarly publications. SSHRC has funded ASP since the Council was founded in 1978. Budget levels have been constant, at about $1.8M, since before the period covered by this evaluation 2008-18. Recent changes to ASP include changes to its assessment system (2013) and a change of name (2014). In 2015, FHSS adopted an OA policy.

The ASP currently offers 180 Publication Grants and five Translation Grants each year for a total contribution of $1.5M dollars. ASP-funded works may be published in print and/or digital formats. ASP statistics are shown in Table 7, below.

Award value and duration

An ASP Publication Grant provides $8,000. Grants are approved in principle before publication and are paid directly to the publisher after publication.

An ASP Translation Grant provides $12,000 to offset the cost of translation and publication. Grants are approved in principle before publication of the translation and are paid directly to the publisher after publication.

Eligibility

Eligible Canadian authors or publishers^{Footnote 45} can apply for an ASP grant to support the publication or translation of scholarly long-form works, such as books and monographs. Authors do not need to have a publisher committed to publishing the work at the time of submission, but they are encouraged to work with a publisher before submission^{Footnote 46}.

Application process and adjudication

ASP staff make preliminary decisions about eligibility for application. If eligibility is not clear, the ASP’s Academic Council makes a formal ruling.

The Academic Council is a six-member committee of scholars. The council advises the Federation on matters relating to ASP’s policies and strategic direction. The council meets at least four times a year to discuss policy issues and make funding decisions on Translation Grant applications.

ASP’s program year is April 1 to March 31. Publication Grant applications are accepted and evaluated throughout the year. Publication Grant applications are assessed by the Publications Committee, which includes specialists in the primary disciplines of the SSH.

In cases where the application is submitted by the publisher, the applicant must provide at least one of the two peer-review reports with the application and may provide both if they wish to do so. The peer-review process must follow ASP guidelines^{Footnote 47} and use the ASP peer-review form. The ASP standard for peer review is single-blind; the identity of the author can be shared with the peer reviewers, but the identities of the peer reviewers must not be shared with the author. For applications by authors, the ASP engages at least two peer reviewers. There is no specific time frame for the peer-review process.

In cases where the ASP engages peer reviewers, once reviews are received, they are forwarded to the author directly if the author is the applicant, or via the publisher if the publisher is the applicant. If the peer reviewer has substantial criticisms of the work, the author must address them within six months.

Complete application packages (including peer reviews and specific sections of the manuscript)^{Footnote 48} are scored by the Publications Committee according to the scale shown in Table 3.

Applications scored within a given calendar month are ranked along with other applications scored in the same month. An allotted number of applications are approved and the remainder are declined. Applications scoring 2.9 or less cannot be approved for funding.

If an application is declined, it may be resubmitted after the work has been revised. An application may only be resubmitted once, and it must be resubmitted within two years of being declined. At least one of the original peer reviewers must provide a detailed report on the revised work.

If the work remains unpublished three years after approval in principle, the grant will lapse and no further application for the grant will be considered. ASP assumes no responsibility for lapsed grants.

Appendix B. Methodology

As this was a joint evaluation for ASJ and ASP, a common approach was used for both initiatives. Common methods were also used wherever possible for both initiatives. Specifics on data collection and analysis methods are provided below. Methods applied only to ASJ and not including data collection and analysis for ASP are not included below but are documented in the evaluation report for ASJ.

Key informant interviews

Key informant interviews were conducted in July and August 2019 as one line of evidence for the evaluation and as part of a sequence of research activities.

Interview participants were selected by the SSHRC Evaluation Division drawing on administrative data for ASJ and ASP to provide a broad cross-section of perspectives. A total of 42 interviews were conducted across eight sub-groups, as follows:

• Editors of journals funded by ASJ (n=8)
• Editors of journals not funded at all or funded first by ASJ in 2018 (n=4)
• Publishers that are frequently funded by ASP (n=7)
• Publishers that are rarely funded by ASP (n=4)
• Academic researchers in the social sciences and humanities (n=6)
• Other stakeholders and representatives of stakeholder organizations (n=6)
• Program representatives (ASJ, ASP) (n=5)
• Government representatives (other federal funders) (n=2)

ASP administrative data analysis

Data were provided by the FHSS on ASP applications (n=2540) between 2007 and 2017. Descriptive statistical analysis was conducted in SPSS by the SSHRC Evaluation Division.

SSHRC CCV data analysis – Canadian SSH researcher publications

A cross-sectional design was used to study researcher publication histories in order to produce estimates of the proportion of the Canadian SSH research population that had published with Canadian journals and book publishers. Full details of procedures followed are available in the Aid to Scholarly Journals Final Evaluation Report.

Document and literature reviews

Four focused reviews of documents and peer-reviewed literature were conducted over the course of the evaluation.

• Relevance review: scholarly literature and funding program documentation related to funding to scholarly publishing in Canada (n=22)
• Performance review: past evaluations and studies specific to ASJ and ASP (n=7)
• Alternatives review: peer-reviewed literature published 2016-May 2019 on government interventions to advance OA publishing (n=69).
• EDI review: SSHRC and tri-agency documents on EDI and ASJ/ASP program documentation related to objectives and program collection of monitoring data (n=8)

Surveys

Three surveys were conducted in Phase 2 of the ASJ-ASPP evaluation, one for each of three populations:

• Canadian SSH journal editors
• Canadian scholarly publishers funded by ASPP
• Canadian SSH researchers who have published through a Canadian scholarly journal or book publisher

The journal editor survey applied to ASJ only. The publisher survey applied only to ASP. The researcher survey applied to both funding opportunities. More information on the publisher and researcher surveys is provided below.

Publisher survey

The population for this survey was defined as representatives at the management/executive level of Canadian scholarly publishers eligible for ASP (N=42). This included both university presses and specialist/generalist presses.

A sampling strategy was not applied. The survey was distributed to one representative of each identified Canadian scholarly publisher eligible for ASP.^{Footnote 49} The survey was soft-launched by Ference & Company to 10 randomly selected participants on February 12, 2020. As no issues were reported, the survey was launched to the remaining participants on February 13, 2020. A reminder email was sent to all recipients who had not already completed the survey on March 3. A targeted reminder was sent to recipients following analysis of response bias on March 6. Telephone follow ups with non-respondents were conducted between March 6 and March 10. The survey remained open until March 13.

Approximately half (49%) of publishers in the survey population fully completed the survey, and a few (n=3) abandoned the survey after starting. Incomplete responses were not included in any analyses. The publishers that completed the survey represent 88% of all ASP funding distributed over the past three years. The margin of error of 16% is considered high for survey data, as it indicates that the survey results are limited in their ability to generalize to the total population. In this case it means that the survey results (which consist primarily of presses regularly receiving ASP funds) are less generalizable to presses that do not regularly receive ASP funds or receive very little ASP funding.

Researcher survey

The population was defined as SSH researchers affiliated with Canadian institutions, who have used Canadian journals or book publishers as a publication channel. This population definition was used to enable a response sample large enough for subgroup analysis of potential differences in the use of Canadian publication channels (e.g., by career stage, language or disciplinary affiliation).

To respect the limited scope of this evaluation, the researcher sample was drawn from two available administrative data sources:

1. Canadian SSH researchers with Canadian publication records in the SSHRC Canadian Common CV (CCV) database. The SSHRC AMIS database was used as a proxy for the population of Canadian SSH researchers in an assessment of representativeness of the CCV data (see above, SSHRC CCV Data Analysis).

   To estimate an appropriate sample size for the survey, a pilot study using the SSHRC CCV data was completed to provide an estimate of the proportion of Canadian academic researchers who publish research using Canadian channels. Based on the results, a sample of 1,500 was drawn from the CCV data using simple random sampling to be representative of the total study population. Individuals with no Canadian affiliation and no publication history were removed, along with researchers who had not published in a Canadian journal or through a Canadian book publisher. This left a sample of 865 researchers.

2. ASP applicants in the years 2016 and 2017. These were included in the sample to ensure sufficient representation of researchers published with Canadian scholarly book publishers. These application years were chosen to allow time for submitted books (successful and not successful) to have been published. Both successful and not successful applicants were included. Applicant data were provided by the FHSS from the ASPP administrative database (n=522): 91 individuals not affiliated with a Canadian institution were removed from list, then a random sample of 50% was extracted (n=215). 27 records were excluded due to duplication or because a valid email address could not be identified (n=188).

The ASPP and CCV lists (n=1053) were compared to identify duplicates before merging the lists. The list was also compared to the list for the ASJ editor survey to ensure researchers would not be surveyed twice. After removal of duplicates, individuals on leave (e.g., sabbatical, parental, medical, unspecified) and one invalid email address, the final sample of n=993 was transferred to Ference & Company for survey administration. The survey was soft-launched on February 14, 2020. The survey was fully launched on February 18, 2020. A reminder email was sent to all recipients who had not already completed the survey on February 24, March 2 and March 6, 2020. The survey closed on March 10, 2020.

Approximately one-third (32%) of researchers in the population sample fully completed the survey. Four researchers responded but were disqualified and 23 did not complete the entire survey; their responses are not included in any analyses. The margin of error of 5% indicates that when generalizing the results of the survey to the population, the range of values above and below the survey value is 5%.

Survey Analysis

Closed-ended responses

Survey data were extracted from the survey platform in MS Excel format. Data were cleaned and verified for analysis in SPSS. Analysis of the closed-ended responses was conducted by Ference & Company.

Significance testing:

Significance testing for variations in survey responses in the researcher survey was undertaken using appropriate tests, including independent samples to test, one-way analysis of variance (ANOVA) and chi-square test of independence.

The publisher survey was conducted with a small, finite population, and all members were included in the frame. As the overall population size and the response sample size were small, descriptive statistics were used exclusively.

Open-ended responses:

Open comment responses were extracted from the survey platform in MS Excel format. Respondent attributes (e.g., unique ID, subgroup membership status) were also extracted. The researcher survey data were then imported into NVivo for analysis by the SSHRC Evaluation Division.

The data for the researcher survey were coded inductively into categories by evaluation question. Once all records had been coded, the contents in the categories were reviewed for internal consistency, distinctiveness from other categories, and to confirm the representativeness of the category label to content held in the category node. Where necessary, content was disaggregated into subcategories to improve internal consistency, categories were merged when insufficiently distinct or underpopulated, and category labels were modified to better reflect the final content set. Where a comparison by subgroup was relevant, a crosstab query was then used to obtain reference and respondent counts (frequencies) by subgroup for each node category. Analysis of open comment responses for the publisher survey was conducted inside Excel using keyword coding by survey question to group responses by category.

Date modified:

2021-06-30