Produced by the SSHRC Evaluation Division
Ference & Company Consulting Ltd.

The Honourable François-Philippe Champagne, P.C., M.P.
Minister of Innovation, Science and Industry

© Her Majesty the Queen in Right of Canada,
represented by the Minister of Industry, 2021

Cat. No. CR22-119/2021E-PDF
ISSN 978-0-660-37302-7

PDF document

---

Table of Contents

• List of Acronyms
• Glossary of Terms
• Acknowledgement
• Executive Summary
  • About the Evaluation
  • Conclusions
  • Recommendations
• 1.0 Introduction
  • 1.1 Evaluation Background and Purpose
  • 1.2 Structure of this Report
• 2.0 Relevance
  • 2.1 Conclusions
  • 2.2 Detailed Findings on Relevance
• 3.0 Performance
  • 3.1 Conclusions
  • 3.2 Detailed Findings on Performance
• 4.0 Cost-efficiency
  • 4.1 Conclusions
  • 4.2 Detailed Findings on Cost-efficiency
• 5.0 Alternatives
  • 5.1 Conclusions
• 6.0 Conclusions and Recommendations
  • 6.1 Conclusions
  • 6.2 Recommendations
• References
• Appendix A—ASJ Program Profile
• Appendix B—Methodology
  • Key Informant Interviews
  • ASJ Retrospective Cohort Study
  • ASJ Administrative Data Analysis (Qualitative Component)
  • ASJ Administrative Data Analysis (Quantitative Components)
  • SSHRC CCV Data Analysis—Canadian SSH Researcher Publications
  • Document and Literature Reviews
  • Literature Review (Government Funding Approaches for OA)
  • Surveys

---

List of Acronyms

• APC – Article Processing Charge
• ASJ – Aid to Scholarly Journals
• ASP – Awards to Scholarly Publications
• EAC – Evaluation Advisory Committee
• ECR – Early Career Researcher
• EDI – Equity, Diversity and Inclusion
• FTE – Full-time equivalent position
• OA – Open Access
• SSH – Social Sciences and Humanities
• SSHRC – Social Sciences and Humanities Research Council
• SSHRC AMIS – SSHRC Awards Management Information System

---

Glossary of Terms

Article Processing Charge. A fee charged to the author to publish an article in open access form.

Canadian research. For the purpose of this evaluation, Canadian research in the context of ASJ has been defined as research conducted by researchers affiliated with Canadian institutions or about Canadian topics (for example, on topics of Canadian geography, culture, history or institutions).

Early Career Researcher (ECR). Academic researchers with five years’ experience or less since their first academic appointment, with considerations for part-time positions and interruptions.

Equity, Diversity and Inclusion (EDI). A tri-agency initiative to increase equity in the workplace for equity-seeking groups. This includes the following four designated groups: women, visible minorities, Indigenous peoples and people with disabilities.

Knowledge mobilization (KMb). SSHRC defines knowledge mobilization as follows: “the reciprocal and complementary flow and uptake of research knowledge between researchers, knowledge brokers and knowledge users—both within and beyond academia—in such a way that may benefit users and create positive impacts within Canada and/or internationally, and, ultimately, has the potential to enhance the profile, reach and impact of social sciences and humanities research” (SSHRC, 2018).

Open access (OA). Open access is defined as availability of the publication online and without charge to the reader. Definitions of subtypes of OA remain fluid. However, a general consistency exists in the literature around the following OA article subcategories.

Gold OA. The full published article is accessible from the journal’s website, immediately on publication and free of charge to the reader, often through fees paid by authors (APCs) or sponsors (Eng, 2017). Some authors assert that an article should also be licensed for reuse if it is to be classed as a Gold OA article (Penn, 2018). However, this is not yet standard practice.

Gold OA articles may be published in journals that are entirely OA (Gold OA journals) or in hybrid journals. Hybrid journals make some of their articles accessible by subscription and others open access, usually in return for payment of an APC.

Delayed OA. The full published article is made accessible from the journal’s website, free of charge to the reader, after an embargo period has passed (Piwowar et al., 2018). Journals that operate entirely on this basis (Delayed OA Journals) generally require a subscription for access to all articles for a set period (e.g. 6 months or 12 months) and make all articles open access after the period ends.

Green OA. Articles are self-deposited, usually by the author, in an open access repository. Such repositories are hosted by a third-party (not the journal), such as an academic institution or other entity providing a disciplinary-based or regional OA repository (e.g., ArXiv) (Borrego, 2016; Bower, 2017).

Grey OA. Articles are made openly available by the author on websites or social media sites that are not repositories, e.g., ResearchGate, Academia.edu or a website of a faculty at an academic institution (Borrego, 2016). Copyright compliance is often not considered (Piwowar et al., 2018).

Research dissemination. Efforts to enhance accessibility and awareness of research findings among potential users. Research dissemination is a component of KMb and concerns activities within the supply side of a KMb system, distinct from uptake or utilization by others. Dissemination is also understood to involve movement of information whereas KMb is more broadly concerned with knowledge (for example, including implementation in practice).^{Footnote 1}

---

Acknowledgement

We would like to thank the Canadian SSH researchers, journal editors, publishers and other sector stakeholders who contributed their time as participants in this evaluation.

We would also like to thank the external members of the Evaluation Advisory Committee for their expert advice and support: Isabelle Bourgeois, PhD, and Stefanie Haustein, PhD, from the University of Ottawa, and Josée Dallaire, Sonia Vani and Gina Hill Birriel from the Federation for the Humanities and Social Sciences.

---

Executive Summary

About the Evaluation

This was a joint evaluation of two funding initiatives: Aid to Scholarly Journals (ASJ) and Awards to Scholarly Publications (ASP). This report presents findings and recommendations for ASJ. A separate report is available for ASP.

The evaluation focused on questions of relevance, performance, cost-efficiency and alternatives. It covered the period 2008-17 for performance-related questions and 2008-18 for questions of relevance, cost-efficiency and alternatives. The following questions were addressed.

1. Is there a need for the federal government to provide direct financial support to journals and publishers in the scholarly publishing sector to increase dissemination of Canadian SSH research results?
2. Do ASJ and ASP objectives align with federal roles and priorities?
3. What contribution has ASJ/ASP funding made to quantity, quality and dissemination of published Canadian SSH research?
4. Are ASJ/ASP delivered in a cost-efficient manner?
5. Are there viable alternative approaches SSHRC should consider to increase dissemination of original Canadian research results in the social sciences and humanities?

A two-phase sequential mixed methods design was used. Phase 1 included analyses of administrative and external datasets, key informant interviews, and document and literature reviews to establish a foundational understanding of the potential influence and relevance of ASJ and ASP. Phase 1 results were tested in Phase 2 with larger samples using surveys and more targeted administrative data analyses and document reviews.

Conclusions

ASJ is one of a small number of initiatives within SSHRC that does not fund individual researchers and research projects but is aimed at addressing underlying constraints in the research system and supporting an environment that enables SSH research. Attention was paid to both intended and unintended outcomes and how they align with SSHRC’s commitments to support and advance SSH research and researchers in Canada. Relevance and alignment were a particular focus due to changes in ASJ and in the surrounding context over time.

Relevance. Canadian journals have served as a publication channel for a majority of Canadian SSH researchers (65%). Canadian journals were rated as important channels for publication by 84% of researchers surveyed: they report that these journals are not only important for the dissemination of their research, but also as content for input to their research, to connect them with peer researchers and target audiences, and for cultivation of Canadian research. Journals eligible for ASJ are operated primarily by volunteer academics on small budgets. Without ASJ, most funded journals would lose more than half of their revenue. Alternate revenue sources are limited, especially for Gold OA journals.

The relationship between SSHRC and Canadian SSH journals is synergistic. SSHRC’s other investments in Canadian research and researchers are augmented by funding to journals that are positioned and mandated to serve Canadian SSH research communities. Taken from another perspective, loss of capacity in Canadian publication channels would likely undermine SSHRC’s other investments, not least research on topics of national importance.

Performance. ASJ funding has enabled a significant and meaningful increase in amount of SSH research published each year. The findings suggest that ASJ also has made small, indirect contributions in some other areas, such as journal capacity to assure manuscript quality, and practices related to accessibility and discoverability of the research published. ASJ’s contribution is notable given its small budget.

Potential. ASJ could provide leverage for SSHRC on other commitments, such as to advance EDI. Canadian journals are positioned to influence EDI at the system level, and some journals already play a role to mobilize research on equity-related topics (e.g., Indigenous research). ASJ is also an important component of SSHRC’s effort to increase quality OA capacity in Canada. The funding is especially important to Gold OA journals whose alternative revenue sources are limited.

Recommendations

1. Continue to offer ASJ funding. At this time, Canadian journals remain important to the SSH research system in Canada. ASJ is demonstrating positive outcomes that would be challenging to reproduce through other means. ASJ is a relatively small investment for SSHRC under a cost-efficient model. ASJ funding is increasingly relevant for journals transitioning to OA.
2. Adjust ASJ’s objectives to align more closely with the Insight Program and ASJ’s key strengths. ASJ’s objectives are under-conceptualized relative to ASJ’s potential within the Insight Program and how Canadian SSH journals are used and valued by researchers. Improved alignment with Insight Program objectives and the role of Canadian journals as identified by researchers (i.e., where these intersect) would improve ASJ’s focus and may enable it to do more with its limited resources. Some specific suggestions are provided in Section 6 of this report.
3. Ensure operational alignment of ASJ to updated objectives. More fully developed objectives should be carried forward into the ASJ adjudication and performance monitoring practices. For example, ASJ adjudication criteria should be reviewed to ensure alignment with and appropriate emphasis on core priorities. Suggestions are provided in Section 6 of this report.
4. Update ASJ’s logic model. ASJ’s current logic model does not accurately capture ASJ’s performance nor the mechanisms by which it contributes to outcomes. The findings reported below provide the basis for an empirically based change model to guide ASJ’s development.
5. At the corporate level, consider using ASJ as a vehicle to further advance key council priorities. The current investment in ASJ is not sufficient to produce more than a very modest contribution to the amount, quality and accessibility/discoverability of Canadian SSH research published. Yet, ASJ has the potential to be a point of leverage for SSHRC in the SSH research system to advance objectives on research excellence, research use, EDI, open research and other areas. Some specific suggestions are provided in Section 6 of this report.

---

1.0 Introduction

This report presents key findings, conclusions and recommendations from a joint evaluation of Aid to Scholarly Journals (ASJ) and Awards to Scholarly Publications (ASP). This report focuses on ASJ. A separate report is available for ASP.

1.1 Evaluation Background and Purpose

About ASJ. ASJ is a funding opportunity within the Social Sciences and Humanities Research Council’s (SSHRC) Insight Program. ASJ’s primary objective is to increase dissemination of Canadian research in the social sciences and humanities (SSH). ASJ seeks to achieve this through funding to Canadian scholarly journals in SSH, recognizing these journals as a primary tool for research dissemination.

ASJ has been active since 1940. It currently provides more than $3M in funding annually to about 160 journals. Funding is awarded through periodic competitions, with the most recent taking place in 2008, 2011, 2014 and 2018. More information about ASJ is available in Appendix A.

About the evaluation. This evaluation’s overall purpose was to inform decisions by SSHRC senior management and program management about the future of ASJ and ASP in a changing environment.

The evaluation was led by the SSHRC Evaluation Division in collaboration with Ference & Company, an external consultancy. It was guided by an Evaluation Advisory Committee (EAC)^{Footnote 2} with oversight by the SSHRC Departmental Evaluation Committee (DEC).^{Footnote 3}

Evaluation questions and scope. Program relevance and performance were identified as priority areas for evaluation, including how the design of ASJ contributes to its performance. Cost-efficiency was also covered as a usual component of SSHRC evaluations.

The evaluation focused on five questions:

The evaluation covered the period 2008-17 for performance^{Footnote 4} and 2008-19 for questions of relevance. Cost-efficiency covered the most recent four fiscal years (2014-19). Question 3’s component on dissemination focused on accessibility/discoverability of publications. Question 5 focused on government funding in support of Open Access (OA) because advancing OA publication is a current key priority for SSHRC.

Evaluation design. A two-phase sequential mixed-methods design was used, under a broad Utilization-Focused (Patton, 2008) and Theory-based (Chen, 2006) approach. Phase 1 was designed to establish a foundational understanding of the potential influence and relevance of ASJ and included the development of a draft theory of change. This phase was needed because of changes to ASJ and in the surrounding context since its last evaluation. Phase 2 was designed to test and refine Phase 1 results to arrive at final findings.

Methods for ASJ. In Phase 1, key informant interviews (n=41) were conducted with program management, journal editors, scholarly publishers, researchers and other sector stakeholders, with 26 of these interviews related directly to ASJ.^{Footnote 5} ASJ administrative data were analyzed (qualitative and quantitative, 2008-19), and the evaluation conducted a retrospective cohort study using data from the Web of Science database and SSHRC administrative data (2008-17, n=194 journals). Reviews of academic literature, working papers and program documentation were also performed in Phase 1.

In Phase 2, the evaluation surveyed lead editors of ASJ-eligible journals (n=286) and SSH researchers (n=993) and conducted focused analyses of ASJ and SSHRC administrative data (2008-18), including financial data and publication histories of a sample of SSH researchers (n=1306). A document review related to equity, diversity and inclusion (EDI) was also performed. In both phases, data from funded journals and similar but not-funded journals were compared where data were available, using naturally occurring comparison groups. Appendix B provides more detailed information on methods.

Methodological strengths and limitations. The principal advantage of mixed-methods designs is the ability to leverage strengths of both qualitative and quantitative methods. For example, limits to generalizability of findings from interviews and limits to context-sensitivity of survey data were offset through synchronization, by using qualitative findings from Phase 1 to inform instrument design and sampling strategy for Phase 2.

Important limitations exist with respect to available data sources. No pre-program data are available. Comparison between funded and not-funded journals relied on naturally occurring groups; differences between the groups likely exist over and above the difference in funding which could not be controlled. Respondents to the evaluation who had been recipients of ASJ funding may overestimate ASJ’s impact, although the evaluation triangulated data from different sources on each indicator to mitigate this. Results should not be considered generalizable outside the populations defined for the study (Appendix B). It is also important to note that the study populations (Canadian SSH journals and researchers) are heterogeneous. For example, SSH has numerous disciplines, each with its own publication practices and influenced by regional and linguistic differences across Canada. ASJ-eligible journals vary in size, maturity, topic focus, readership and language of publication. As such, aggregate results may obscure important variation at subgroup level. Although subgroup-level analysis was conducted where possible (e.g., language groups), this was not always possible due to small subgroup sizes in the samples.

1.2 Structure of this Report

Sections 2 to 5 of this report present key findings organized by focus area: relevance, performance, cost-efficiency, alternatives. Section 6 presents overall conclusions and recommendations. Appendix A provides a profile of ASJ. Appendix B provides details on methodology.

---

2.0 Relevance

The evaluation assessed relevance of ASJ in terms of the importance of domestic journals to Canadian SSH researchers, the extent to which ASJ is critical to journal viability, relevance as identified in academic literature and by other federal agencies, and alignment of ASJ objectives with SSHRC priorities.

2.1 Conclusions

ASJ is relevant to Canadian SSH researchers. A majority of Canadian SSH researchers (65%) have published through a Canadian journal. Of those responding to the survey, 84% reported Canadian journals to be important (moderately to essential) for publication of their research.

More than half of researcher survey respondents described Canadian journals as essential or very important for connecting with other researchers in their field and for source material as an input to their research. In terms of researchers’ career advancement, Canadian journals appear to be very important for some researchers (41%), but not others (25%). Data across lines of evidence are consistent that Canadian journals are more important to some researchers than to others, particularly―but not exclusively―those who conduct research about Canada.

When selecting a journal, the journal’s content focus, audience and peer reputation outweighed other considerations, including impact factor or open access options. Researcher responses on valued characteristics of journals aligned with how journal editors described their mandates to support research communities.

ASJ is important to journal viability. Results from multiple lines of evidence indicate that ASJ is critical to the survival of a minority of funded journals, with differential importance for some subgroups. For the majority of journals, the impact of loss of ASJ funding would be in reduced capacity. Without ASJ, most funded journals would lose more than half of their revenue. Alternate revenue sources are limited, especially for Gold OA journals.^{Footnote 6} Estimated three-year survival of journals by their editors shows a 21% difference between funded and non-funded journals, based on current finances. Among funded journals, French-language respondents appear to perceive greater vulnerability to a loss of ASJ. ASJ makes a significant contribution to human resource capacity of funded journals in production-related areas. A small carryover effect of ASJ to journals’ capacity to engage in activities related to journal strategy and development is indicated, although the evidence is not conclusive.

Alignment of ASJ objectives with federal and SSHRC priorities. Other federal programs and academic literature provide rationales for providing funding to domestic publishing channels, which also reflect SSHRC priorities. However, ASJ’s objectives are broadly stated, limiting assessment of alignment on this basis.

ASJ’s objectives and monitoring data do not incorporate EDI; however, the findings suggest potential alignment among journal activities and SSHRC’s EDI objectives, and potential for ASJ to support advancement of EDI in the research system via support to journals.

2.2 Detailed Findings on Relevance

Importance of domestic journals to Canadian SSH researchers

An estimated 65% of Canadian SSH researchers have published in a Canadian journal (95% CI: 62.3 to 68.4)^{Footnote 7} based on an analysis of the Canadian Common CV dataset conducted for this evaluation (see Appendix B). Among those responding to a survey for this evaluation (n=314),^{Footnote 8} 83% reported that Canadian journals in their field are important for the publication of their research; 57% rated them very important or absolutely essential (Figure 1).

Figure 1 long description

Figure 1. Importance of Canadian journals for publication of research, SSH researcher ratings (n=314)

Category	Percentage
Absolutely essential or very important	57.3%
Moderate importance	25.7%
Little or no importance	16.0%

About half of respondents also rated Canadian journals in their field as absolutely essential or very important as source material for their research (53%) and for connecting them with others in their field (51%). When asked about the importance of Canadian journals for their career progression, researchers gave a more mixed response: 41% reported Canadian journals to be essential or very important; a further 24% reported that they are of little or no importance (Figure 2).

Figure 2 long description

Figure 2. Researcher ratings of the importance of Canadian journals in their field (n=314)

	Career progression	Connecting with others	Source material	Publication
Absolutely essential or very important	41%	51%	53%	57%
Moderately important	32%	28%	32%	26%
Of little or no importance	24%	19%	13%	16%

No statistically significant differences were found in the above ratings by language, career stage or disciplinary category (humanities vs social sciences), except with respect to the importance of Canadian journals for connecting with others in their field. In this respect, early career researchers rated Canadian journals higher in importance than established researchers;^{Footnote 9} English-speaking researchers rated them higher than Francophone respondents.^{Footnote 10}

Respondents who reported Canadian journals to be important for the publication of their research (n=260) were asked for additional comments. About two-thirds provided a detailed response (n=165), with some describing the importance of Canadian journals for reaching a specific target audience (42%) and some describing the fit between their research and the focus of these journals (30%). For example, respondents described research focused on Canadian topics to be less relevant outside of Canada and they described the importance of Canadian journals for the development of some fields of research within Canada. About 10% (n=23) described the quality and reputation of the journals, with a few responding that Canadian journal(s) are among the most widely read journals in their field worldwide.

Survey respondents were also asked about factors influencing their selection of a journal for manuscript submission (not necessarily a Canadian journal). Similar to responses above, the content focus of the journal, its target audience and the journal’s reputation with peer researchers were reported to be essential or very important by over 80% of respondents (n=314; Figure 3). These top three factors were consistent across language, career stage and disciplinary category. Mixed responses were received on other suggested reasons for journal selection, such as impact factor and OA options. Most respondents (71%) rated bilingual publication to be of little or no importance when selecting a journal for a new manuscript submission (Figure 3)^{Footnote 11}.

Figure 3 long description

Figure 3. Drivers of choice of journal for new submission, researcher ratings (n=314)

	Not important at all	Of little importance	Very important	Absolutely essential
Content focus of journal	0%	0%	41%	54%
Target audience of journal	0%	1%	55%	29%
Journal reputation with peer researchers	1%	1%	53%	29%
Likelihood of the article being accepted	2%	10%	34%	7%
Journal impact factor	7%	14%	28%	10%
Estimated time to publication	4%	15%	27%	9%
Open access options available	12%	19%	21%	9%
Bilingual publication	39%	32%	5%	1%

Respondents explained that the fit between their research and the content focus of the journal is important to ensure that peer reviewers will be selected who understand the research and that the article will reach the desired audience. A distinction was made in terms of size vs nature of audience. A large audience was described as important for some (e.g., “a global audience”) and a targeted audience for others (e.g., other scholars in a subspecialty, front-line professionals in an area of practice).

In sum. A majority of Canadian SSH researchers have published in Canadian journals and more than half of those surveyed reported the Canadian journal(s) in their field to be absolutely essential or very important for publication, connection with peers and as source material. A substantial proportion (41%) also reported them to be essential or very important for their career progression.

Importance of ASJ to journal viability

Phase 1 results^{Footnote 12} suggest that most journals would survive the loss of ASJ funding, although the loss of funds would reduce survivors’ capacity and may affect journals’ long-term viability. Some journals depend on ASJ for basic operation; these journals would likely cease to exist if funding was lost.

Short-term survival. In phase 2, share of ASJ funding to total journal revenue and journals’ options for alternative revenue were considered. The survey also compared funded to non-funded editor projections for journal survival in the short term.

Journal revenue. On average, ASJ funds represent 67% of the overall revenue of funded journals, as reported by funded editors in the survey (n=68). A majority (64%) report ASJ to be more than half of their journal’s annual revenue. In 2018, the average ASJ grant was $26,363/journal/year, indicating that the total revenue of most funded journals is less than $53,000/year.^{Footnote 13}

Alternate revenue sources. For both funded and not-funded journal editors, the most frequently reported other revenue sources were subscriptions and royalties. Fewer than 40% of survey respondents reported contributions from an academic institution; fewer than 20% reported contributions from scholarly societies or professional associations; 13% of funded editors (n=9) indicated that they had no source of revenue other than ASJ (Figure 4). Only six journals reported using Article Processing Charges (APCs) to publish OA, only one of which is a funded journal. The use of APCs is revisited below in Section 3 (Performance).

Figure 4 long description

Figure 4. Alternate revenue sources over past 12 months, editor survey respondents (n=132)

	Not currently funded	Currently funded
	(n = 64)	(n = 68)
No sources other than ASJ	0%	13%
Contribution from a professional association	11%	4%
Other	8%	16%
Contribution from a scholarly society	19%	13%
Provincial government funding	6%	26%
Contribution from an academic institution	36%	38%
Royalties	52%	43%
Subscriptions	55%	51%

Provincial funding was reported by 26% of funded respondents, all of whom lead French-language or bilingual journals.^{Footnote 14}

ASJ appears to be relatively more important for Gold OA journals. ASJ funds to total revenue is highest among Gold OA journals (average 78%). Delayed OA and Hybrid/Non-OA both reported significantly lower share of revenue from ASJ (54% and 59%, respectively; p < .001).^{Footnote 15} Likewise, for journals without subscription revenue, ASJ represents a significantly larger share of average revenue (73%) compared to those receiving subscription revenue (52%; p<.001).^{Footnote 16}

Editor projections on journal survival. In the survey, all editors were asked about the likelihood of their journal being in operation in three years, given their current financial situation.^{Footnote 17} While a majority from both funded and not-funded groups responded that their journal would definitely or very probably still be in operation in three years, a significantly higher proportion of funded editors (75%) gave this response than not-funded editors (54%), a difference of 21% (p=.005; Figure 5).^{Footnote 18}

Figure 5 long description

Figure 5. Projected 3-year survival of funded versus not funded journals, editor responses

		Probably or definitely not	Possibly or probably	Definitely or very probably	Total
Not funded n=59	% within 2018 funded versus not funded	12%	34%	54%	100%
Funded n=67	% within 2018 funded versus not funded	6%	19%	75%	100%

In a second question, the editors of ASJ-funded journals were asked to estimate the likelihood of their journal being in operation in three years without ASJ funding. About 43% changed their projection from positive (with ASJ funding) to negative. A higher proportion of French-language editors projected they would definitely or probably not survive without ASJ (76%), compared to editors of bilingual journals (59%) and English-language editors (32%).^{Footnote 19}

In sum. Results from multiple lines of evidence indicate that ASJ is critical to the survival of a minority of funded journals, with differential importance for some subgroups. Without ASJ, most funded journals reported they would lose more than half of their revenue. Alternate revenue sources are limited, especially for Gold OA journals. There was a 21% difference in estimated three-year survival between funded and non-funded journals based on current finances. Among funded journals, French-language respondents appear to perceive greater vulnerability to a loss of ASJ.

Long-term viability. Phase 1 findings^{Footnote 20} suggested that ASJ funding is also important for longer-term viability. These data suggested that ASJ funds increase journal capacity to maintain publication levels and may allow for volunteer editors to consider strategy and development (e.g., adoption of technology). In Phase 2, reported capacity was compared between funded and not-funded journals in areas of production, strategy, adaptation to change and mandate fulfillment.

Human resource capacity. Based on Phase 2 survey data, most SSH scholarly journals (80%) have no full-time employees. Nearly all editors (95%) reported using volunteer labour. Funded and non-funded editors reported a similar number of volunteers (5-7 volunteers per journal, excluding peer reviewers). However, among journals funded by ASJ, 100% of those funded continuously since 2008 reported having at least some paid support, as did 87% of journals funded in some but not all years in this period. In contrast, only 50% of non-funded journals reported having any paid support.

Funded editors also reported a higher number of part-time positions and hours worked per paid position than non-funded editors. Journals funded continuously by ASJ since 2008 reported, on average, 0.68 total full-time equivalent (FTE) paid positions, compared with 0.30 FTE average reported by non-funded journals.^{Footnote 21} Total FTE positions differed significantly by funding status (p = .037), although post-hoc comparisons between funding groups were not statistically significant.^{Footnote 22} The mean difference of 0.38 FTE between continuously funded and non-funded journals (about 1/3 of a full-time position) represented 592 additional staff hours per year for funded journals.^{Footnote 23} A substantial number of paid positions were held by students (average 49% for funded journals, 38% for not-funded journals).

In addition to paid positions and volunteers, editors also reported paid contracting activities: 72% of continuously funded editors reported contracting at least one type of activity over the previous 12 months, compared with 62% of intermittently funded and 35% of not-funded respondents. This difference between groups is statistically significant (p=.020).^{Footnote 24} By language, 70% of French-language journals reported contracting at least one activity, compared with 49% of English-language journals.

Production activities were by far the most often reported type of contracted activity, including copyediting (71%), layout/graphic design (48%) and translation (43%). Bilingual and French-language journals reported contracted translation more often than primarily English-language journals. Dissemination-related activities were also reported (e.g., web hosting) (13%).

The importance of funds for production-related activities is consistent with findings from analysis of application data across the period 2008-18 (n=60), although these data also indicate that a major use of ASJ funds over the decade was related to technology (see Section 3, below).

Capacity for strategic development. Funded and non-funded editors were asked about their human and financial capacity to make progress toward strategic goals, respond to changing conditions in the sector and fulfill their mandate.

Mandate fulfilment. Editors described journal mandates mostly in terms of building and sustaining communities of researchers: for example, researchers in a specific discipline or subdiscipline (49% of responses), in a field that crosses disciplines (e.g., in social justice) (27%), researchers working on Canadian or regional topics (e.g., Canadian law, Quebec history) (26%), or on research for a specific population (e.g., LGBT, Indigenous groups) (7%). A majority of editors in all groups reported sufficient capacity to act in accordance with their mandate (Figure 6), suggesting that ASJ has minimal or no impact in this area.

Journal development and adaptation. Differences between groups in reported capacity for journal development were not statistically significant. However, they did vary by level of funding in the hypothesized direction, with funded journals consistently reporting greater capacity.^{Footnote 25} In general, editors in all funding groups were more likely to report sufficient human capacity than financial capacity, possibly because of their similar volunteer base (see above). In terms of financial capacity, a minority of non-funded editors reported sufficient capacity to make progress toward strategic goals (35%) and to respond to changing conditions in the sector (34%), compared with a majority of continuously funded editors (64% and 60%, respectively). Results for financial capacity are shown in Figure 6.

Figure 6 long description

Figure 6. Financial capacity for journal strategy and development, % of editors responding that they “completely” or “mostly” agree they have sufficient financial capacity (n=132)

	Capacity to respond to changing conditions in the sector	Capacity to make progress on strategic goals	Capacity to fulfill mandate	n
Non-funded	34%	35%	68%	25
Intermittently funded	43%	53%	75%	81
Continuously funded	60%	64%	86%	26

In sum. ASJ makes a significant contribution to paid human resource capacity of funded journals in production-related areas. This likely explains higher performance of funded journals in terms of the amount of research published (see Section 3, below). A small carryover effect of ASJ on journals’ capacity to engage in activities related to strategy and development is suggested but the findings are not conclusive.

ASJ’s objectives are listed in Appendix A. In brief, ASJ has sought to increase dissemination of and access to SSH research. Since 2008, the program has supported journals to transition to digital formats. From 2018, ASJ objectives included assisting journals to take advantage of digital technologies and innovative practices and to encourage SSH journals to transition to OA.

Alignment of ASJ’s objectives with federal roles and priorities was assessed with reference to federal policy, position and program documents, as well as academic studies on outcomes of funding to domestic journals. Attention was paid to SSHRC’s commitment to equity, diversity and inclusion (EDI).

Alignment of ASJ with federal roles and priorities, reflected in government and academic literature

Federal policy and position documents generally use broad language, and review of these documents found no specific references to a need to provide direct support to scholarly journals. At the program level, two other federal agencies―Canadian Heritage (CH) and Canada Council for the Arts (CCA)―state rationales for funding Canadian publications: these relate to ASJ. These were described by CH and CCA representatives interviewed for the evaluation (n=2) and articulated in program documentation: Canadian publications are important but not always cost-effective due to Canada’s small market (CH) and publishers are a mechanism through which organizational goals can be met because of their capacity to advance work of authors in the field (CCA).

Research literature on government funding to domestic scholarly journals^{Footnote 26} reports benefits in two primary areas: service to researcher communities and dissemination of research. For example, domestic journals are reported to cultivate local research communities across institutional silos and to “incubate” research content. Domestic journals may also provide quality publication channels in languages other than English. Where non-English publication options are limited, this can reduce pressure on non-English speaking researchers to publish in English or use predatory journals and may facilitate knowledge exchange with local non-academic practitioners working in minority languages. Where funding supports translation, this is reported to increase the reach of research outputs to international audiences. Lastly, government funding of domestic journals is proposed to aid the development of quality, equitable OA options^{Footnote 27} (e.g., with no or low APCs) and journal capacity to adopt practices to increase discoverability of domestic research.

In sum. Other federal programs and academic literature provide rationales for providing funding to domestic publishing channels, some of which reflect SSHRC priorities. In comparison, ASJ’s objectives are broadly stated and of limited use for understanding how they align with funding under SSHRC’s mandate.

ASJ, EDI and Official Language Minorities

SSHRC has a commitment to strengthen EDI in the SSH research system, including to promote the integration of EDI-related considerations in research design and practices and to increase equitable and inclusive participation in the research system.^{Footnote 28} SSHRC also has commitments to Official Language Minority Communities. ASJ’s alignment with these priorities was assessed through review of program documents and surveys of editors and researchers.

ASJ’s stated objectives do not currently reflect SSHRC’s EDI priorities, although its objectives are not contradictory. The absence of reference to EDI may reflect the years in which ASJ has been active before EDI priorities were established. ASJ does not currently collect EDI-related monitoring data. As this funding is directed to organizations rather than individuals, an appropriate level for additional data collection would need to be carefully considered.

The key role played by journals, editors and reviewers to influence EDI in research practice and publication is well recognized (e.g., Henrich et al., 2010). In ASJ’s context, qualitative data^{Footnote 29} suggest that some funded journals pursue an explicit mandate to advance knowledge of Canada’s diverse population.

Editors were also asked in the survey if their journal had undertaken any activities explicitly aimed at increasing equitable and inclusive participation in scholarly publishing in the past 12 months. Significantly more continuously funded editors (72%) reported that they had undertaken such activities than non-funded participants (46%; p=.013).^{Footnote 30}

Editors were asked to describe these activities. The majority (80%) reported efforts to increase the diversity of the journal’s editors, including gender diversity, and seeking greater representation by geography, language, sexual orientation and/or racialized groups. Efforts to increase or sustain a diversity of topics were also reported (34%), such as special theme issues on topics related to EDI. The third major response category related to diversity of authors, described by editors in terms of geographic diversity, greater participation by Francophone authors, racialized authors, emerging scholars and others.^{Footnote 31} A few journals reported efforts to expand the diversity of their peer reviewers (n=5). A small number described the mandate of their journal as equity focused, with all activities undertaken through an equity lens (n=6).

Official languages

A related priority under the Official Languages Policy is to promote linguistic diversity and inclusion.

In the survey of journal editors, they were also asked about language barriers they may have encountered in the process of applying for ASJ funding. On a scale of 1 to 7 with 1 as “absolutely no barriers” and 7 as “significant barriers,” 98% of the 84 respondents reported 1 or 2 on the scale.

In the survey of researchers, respondents were asked if they had experienced any language barriers with respect to publishing their research. Of those responding in French, 61% (n=79) reported experiencing at least some barriers, compared with 9% of those responding in English (n=235).^{Footnote 32} Researchers who reported experiencing language barriers were asked to describe the nature of the barriers faced. Those who replied (n=61) described various experiences, including: pressure to publish in English due to lower value placed on French-language publications at their institution (n=15); bearing the cost of translation to publish in English (n=14); and reduced productivity and more critical peer reviews due to publishing in a second language (n=14). Of note, although 23% of survey respondents reported French as their primary language at work, 80% reported English when asked which language they typically used for book and article manuscripts. Only 12% of survey respondents reported French and 8% responded “it depends.”

Notably, when asked about characteristics of journals influencing their choice for submission of a new manuscript, French-speaking respondents rated bilingual publication by the journal higher than English-speaking respondents^{Footnote 33} (p < .001). Overall, only 20% of French-speaking respondents indicated that bilingual publication was essential or very important to them when selecting a journal, possibly because of the proportion that typically publish in English.

In sum. ASJ is unusual for SSHRC as SSHRC generally funds individual research projects, with fewer initiatives directed at organizations. While points of alignment are evident between ASJ and SSHRC priorities, including for EDI, these are not explicitly stated in ASJ’s objectives. The findings suggest potential for ASJ to support advancement of EDI and official languages in the research system. This could become a performance-related area for ASJ in future.

---

3.0 Performance

The evaluation assessed performance on three potential outcomes: contribution to the quantity, quality and accessibility/discoverability of published SSH research.

3.1 Conclusions

Overall, ASJ appears to have made a small but significant contribution to SSH research publication, proportionate to the small size of the initiative.

Quantity. On average, ASJ-funded journals published about 27% more articles per year than similar non-funded journals. ASJ’s impact on the amount of research published is likely due to its relatively small but significant contribution to the production capacity of funded journals (see Section 2, above).

Quality. The findings suggest a limited contribution by ASJ to the quality of published research. This is an indirect contribution via increase in the capacity of journals to sustain important quality processes.

Accessibility and discoverability. As with quantity and quality, any contribution by ASJ to dissemination is indirect, i.e. through incentives and support to funded journals. Change in the use of ASJ funds over the decade demonstrate efforts by journals to respond to a series of developments in technology. A small but consistently higher number of international submissions to funded journals provides some indication of higher global reach associated with funding. Finally, differences between funded and non-funded journals in adoption of some discoverability practices suggest influence of ASJ in specific areas. Overall, the data indicate that ASJ has made a limited contribution to the dissemination of SSH research.

3.2 Detailed Findings on Performance

Contribution to Quantity

To assess ASJ’s potential contribution to the amount of SSH research published in Canada, a retrospective cohort study was conducted to compare publication rates of funded and non-funded journals (n=194), using data from the ASJ administrative dataset and Web of Science. Journals funded by ASJ in the period 2008-17 produced, on average, 27% more articles annually than similar journals not funded through ASJ, 95% CI [8.0, 49.2] p = .004. This analysis controlled for journal discipline, language and frequency of publication.

As reported in Section 2, above, ASJ funding provided additional capacity for production-related activities of funded journals. This is consistent with analysis of ASJ application data from 2008-18 (n=60) in which applicant editors described the ability to pay for some production-related support as a primary contribution of ASJ funding. Production capacity was identified by two-thirds of both funded and non-funded editor participants in interviews (n=8) as a primary constraint on publication.^{Footnote 34} All funded interview participants (n=8) reported that without ASJ funding, if their journal survived, they would have to reduce the number of articles they could publish in a year. In sum, multiple lines of evidence support a significant contribution by ASJ to the amount of SSH research published, via increased production capacity to funded journals. The data also suggest that this increased capacity is relatively small (see Section 2, above) and that capacity remains a constraint on publication, even for funded journals.

Contribution to Quality

Phase 1 results^{Footnote 35} suggested two mechanisms through which ASJ might contribute to the quality of published SSH research. One is through the process of selecting journals for ASJ funding; the other by increasing the capacity of funded journals to sustain or improve quality assurance processes.

Selecting journals with strong quality processes. ASJ’s adjudication process was reviewed to assess whether journals’ quality assurance processes related to success in ASJ competitions. In other words, the evaluation sought to establish whether ASJ’s selection procedures systematically advantaged journals with the highest internal quality processes and, in this way, enabled ASJ to contribute to the quality of published SSH research, as proposed by interview participants. ASJ’s review procedures as currently designed, however, are not best suited to support assessment and selection of journals on this basis.

ASJ adjudication committee members are asked to score each applicant journal based on three major criteria.^{Footnote 36} One of the three, “Quality and Impact of the Journal,” addresses quality. Five subcriteria are provided to help the committee arrive at an overall score. Of these five, one relates to the journal’s manuscript selection and peer-review process.

A quantitative analysis of the relationship between score on this subcriterion and journal success rate was not possible because the five subcriteria were considered by committee members in a holistic manner to produce the overall “Quality and Impact” score. The degree to which this subcriterion influences applicant scores is therefore not known. The ASJ adjudication process is not designed to consistently select for and advantage journals on this basis, however. There is just one relevant subcriterion among five under “Quality and Impact.” This, in turn, is only one of three major criteria contributing to the overall score of the application. Consistent differentiation among applicants by this subcriterion is also challenged by differences in information provided in applications. In sum, a contribution by ASJ to the quality of published research via this pathway could not be established.

Contribution to quality via capacity. All interviewed funded editors (n=8) reported that ASJ funding does not directly influence the quality of the research they publish. They reported a strong intrinsic commitment to quality. While this commitment is not related to funding, they proposed that ASJ contributes to journals’ capacity to sustain quality processes―peer reviews in particular―and other quality control processes such as plagiarism checks, reference checks and copyediting, and that funded editorial teams have more time to work with authors to develop their manuscripts.

In the survey, funded and non-funded journal editors were asked to indicate the extent to which their journal has had the necessary capacity to carry out quality processes, for example, to ensure all manuscripts receive high quality editorial direction and to identify peer reviewer(s) that are appropriately specialized in each manuscript’s subject matter. Regardless of funding status, a strong majority of editors indicated that they had sufficient human resource capacity for most quality processes (>70%), possibly because of the journals’ similar volunteer levels. Responses were also similar by language of journal. However, funded editors were consistently more likely to report sufficient financial capacity for quality processes compared to non-funded editors. The response by continuously funded editors was 19-30% higher (Figure 7).

Figure 7 long description

Figure 7. Proportion of editors completely or strongly agreeing that their journal has sufficient financial resources to address quality processes, by funding status (n=132)

	Ensure a timely review for all manuscripts*	Identify potential peer reviewers that are appropriately specialized in each manuscript’s subject matter*	Ensure the peer review process is carried out to a high standard for all manuscripts	Ensure all manuscripts receive high quality editorial direction	Apply its quality assurance processes consistently to all manuscripts
Non-funded (n=26)	38%	50%	65%	2%	62%
Intermittently funded (n=81)	58%	68%	74%	79%	78%
Continuously funded (n=25)	68%	80%	84%	84%	84%

*Statistically significant

The difference in response is statistically significant in two areas: capacity to identify peer reviewers appropriately specialized in each manuscript’s subject matter (p<.01), and capacity to ensure a timely review for all manuscripts (p=.01).^{Footnote 37} Of note, only half of non-funded editors reported sufficient capacity to identify appropriately specialized reviewers and only 38% reported sufficient capacity for timely review.

Notably, qualitative comments in both editor and researcher surveys identified peer review as an area of concern. In the editor survey, a substantial number of respondents (n=23) commented about the resource intensiveness of a strong peer-review process and increasing scarcity of peer reviewers. In the researcher survey, a substantial number of respondents (n=21) provided descriptive comments about the importance of journal quality with reference to the quality of peer review and editor guidance, including in terms of constructive feedback and timeliness/appropriateness of peer reviewers (i.e., expertise in the area).

Submissions as an indicator of journal quality. Domestic (Canadian) submissions were compared for funded and not-funded applicants using data in ASJ administrative data from 2008-14 (n=291).^{Footnote 38} Applicants with continuous funding consistently reported more domestic submissions than both intermittently funded and non-funded applicants over this period, with an average difference in 2014 of 20 submissions/journal over non-funded journals (1,825 submissions total). Continuously funded journals also reported more stable submissions than the other two groups over the period.^{Footnote 39} This provides some support to an association between journal quality as perceived by researchers and continuous ASJ funding, although only to the degree that journal quality influences submission decisions (see Section 2, above).

In sum. The current design of ASJ does not enable it to contribute directly to the quality of published research. However, the findings suggest an indirect contribution, i.e., that it has a small but significant impact on capacity of funded journals to sustain certain quality processes, most notably quality peer review. Qualitative feedback suggests quality peer review (e.g., recruiting reviewers with relevant expertise, securing timely and constructive feedback) is particularly resource intensive for journals, but very important to researchers.

Contribution to accessibility and discoverability of Canadian SSH research

In the period 2008-17, ASJ provided incentives for journals to take advantage of innovations in the sector, including technological advances in digital publication, with the objective to increase dissemination of Canadian research. In 2018, ASJ introduced a requirement for phased OA adoption. The evaluation considered the possible contribution of ASJ to digital transition of funded journals in the period after 2008, as well as the current use by journals of various practices related to accessibility and discoverability of research. The latter include factors important in the Canadian context (e.g., bilingualism) and other factors drawn from a framework developed by the Scholarly Publishing and Academic Resources Coalition (SPARC) and Public Library of Science. This framework is intended to support assessment of open publication by considering aspects such as readership, reuse, copyright, posting, and machine readability (SPARC, 2014). Some information is also provided below about response to changes to ASJ in 2018. As these changes are very recent, this information is for benchmarking purposes only.

Digital transition. Administrative data indicate that most applicants to ASJ were already publishing in both print and digital form in 2008. The proportion of digital-only journals increased in the funded journal pool from about 8% in 2008 and 2011 to about 33% after 2014.

Analysis of a sample of successful ASJ applications from 2008-18 (n=60) provided a longitudinal description of change in the use of ASJ funds. While print-related use of funds (e.g., postage) was common in 2008, this virtually disappeared over the decade. Digital publication activities were increasingly reported from 2008 forward, with funds before 2014 used primarily for transition, for example, hardware for digitization of print archives, software to convert from mail-based to e-payment systems, and initial development of digital workflow processes. From 2014, use of funds for upgrades were more common, for example to enhance user search tools of digital archives or to meet updated web hosting requirements. A primary theme in the data from 2014 onward was the successful transition to digital workflows without disruption of publication schedules, including conversion of journal archives (often decades of archives) to be fully searchable, with editors identifying the importance of ASJ support.

International submissions. International submissions were analyzed as an indicator of global reach, by comparing submissions to funded and non-funded journals as reported in ASJ applications from 2008-14.^{Footnote 40} International submissions to continuously funded journals (n=53) were consistently higher on average than to other journals over the period.^{Footnote 41} A small but steady increase in submissions was reported by applicants funded in prior competitions, both continuously and intermittently. Non-funded journals did not report a similar increase. In 2014, the average number of international submissions for continuously funded journals (n=53) was 54.8% higher than for non-funded journals (n=28).^{Footnote 42} Of note, ASJ administrative data indicate that most journals applying to ASJ after 2008 were either hybrid (print and digital) or digital only, including the non-funded journals. Therefore, differences in international submissions were not likely due to digital vs print status per se, but to other factors influencing reach.

Adoption of practices for content discoverability. In the survey of journal editors, funded and non-funded editors were asked to identify if their journals had used certain practices to help readers find published articles, and related to author copyright and self-distribution (Figure 8). It is important to note that in the 2008-17 period, ASJ did not provide directed incentives for specific discoverability practices, but rather offered support for dissemination, digital transition and innovation generally. While the detailed results below are useful for forward benchmarking, from a performance perspective these data should be taken only as an overall indicator of journal innovation in accessibility/discoverability.

Figure 8 long description

Figure 8. Proportion of editors reporting use of discoverability/accessibility practices in the prior 12-month period, by funding status (n=132)

	Use of international standard author identifiers (e.g., ORCID ID)	Targeted outreach to non-academic audiences	Allowing article text to be API accessible and/or machine readable	Allowing article metadata (e.g., citation data) to be API accessible and/or machine readable	Translation of abstracts and/or full text free of charge for bilingual publication	Automatic posting of articles in trusted third-party repositories	Use of international standard article identifiers (e.g., DOI)	Active promotion of published articles (e.g., via social media, journal website, scholarly collaboration networks)	Maintaining a system for preserving access to articles in the event that the journal is no longer published	Indexing in international directories (e.g., DOAJ, Project MUSE…)
Non-funded (n=26)	38%	31%	8%	23%	27%	50%	65%	88%	65%	88%
Intermittently funded (n=81)	38%	38%	35%	51%	58%	51%	78%	88%	74%	90%
Continuously funded (n=25)	24%	40%	44%	52%	56%	56%	76%	84%	84%	88%
Difference	-14%	9%	36%	29%	29%	6%	11%	-4%	19%	0%

For most practices, regardless of funding status, reported adoption was high for some (e.g., indexing in international directories) and low for others (e.g., use of standard author identifiers), suggesting no association with ASJ funding in these areas. However, significant differences were found between funded and non-funded journals on four items: maintaining a system for preserving articles, translating abstracts and/or full text free of charge, allowing article text to be API accessible and/or machine readable, and allowing article metadata (e.g., citations) to be machine readable.^{Footnote 43}

Funded editors who reported these practices were asked about the likelihood that they would have adopted the practice without ASJ funding. Figure 9 shows responses of adopters funded continuously by ASJ, where at least 50% of respondents attributed adoption to ASJ. Three of these are practices for which significantly more funded editors reported adoption than did non-funded editors. Group sizes are small, however, and these data should be taken only as a small additional support for a role of ASJ funding in adoption of these practices.

Figure 9 long description

Figure 9. Likelihood of adoption of practices without ASJ funding, % continuously funded editors

	Use of international standard author identifiers (e.g., ORCID ID) (n=19)	Translation of abstracts and/or full text free of charge for bilingual publication (n=14)	Allowing article metadata (e.g., citation data) to be API accessible and/or machine readable (n=11)	Allowing article text to be API accessible and/or machine readable (n=13)
Probably not or definitely not	50%	50%	54%	63%
Possibly – definitely	50%	50%	46%	37%

In the survey, editors were also asked if their journal allows the published version of articles to be deposited in an open access repository by the author without contravening copyright (i.e., Green OA), and if authors retained copyright for the articles published in their journals, including reuse rights. Fewer than half reported allowance of deposit immediately following publication (45% overall), although a further 27% reported allowance after an embargo period. Non-funded editors were more likely to report allowance of immediate deposit (54%) than continuously funded editors (28%). A majority in both funding groups reported that authors retain copyright (65% of non-funded and 60% of funded editors).

In sum. ASJ has sought to support Canadian SSH journals to innovate and adapt in a period characterized by rapid change in dissemination methods, and thereby contribute to dissemination of SSH research. Change in how ASJ funds were used over the decade reflects a response by journals to a series of advances in digital publication technology. As noted in Section 2, above, a majority of continuously funded editors (60%) reported sufficient financial capacity to respond to changing conditions in the sector, compared with a minority of non-funded editors (34%). A small but consistently higher number of international submissions to funded journals provides some indication of increased global reach. Finally, significant differences in adoption between funded and non-funded journals on specific discoverability practices, including free translation and machine-readability of text, suggest influence of ASJ. Overall, the data provide some evidence of ASJ contribution to the dissemination of SSH research.

Changes to ASJ in 2018 to encourage OA

ASJ implemented a requirement in 2018 for phased transition to OA, along with incentives for use of a digital publication platform. These changes are too recent for outcomes to be assessed. However, information related to these changes is provided below for benchmarking purposes.

Adoption of open access models. Respondents to the editor survey were asked about the OA model currently used by their journal.^{Footnote 44} Most journals funded continuously during the 2008-18 period (92%) reported using either Gold or Delayed OA, compared with 77% of non-funded journals and 55% of those funded in some competitions but not others. The difference between continuously and non-funded journals is very small (15%) but does indicate that long-term ASJ funding has not inhibited OA adoption, as has been reported for funding initiatives to domestic journals elsewhere (Björk, 2019). Further, when compared with other journals in their age group, a significantly higher proportion of continuously funded journals reported OA status (Gold or Delayed): 92% (n=22) vs 35% (n=67), p<.001.^{Footnote 45}

Figure 10 illustrates differences in access models reported by age group, with younger journals (established after 2000) more likely to be Gold OA. Journals older than 20 years are more diverse. Almost all (96%) of ASJ’s continuously funded journals are in this second, older, group.^{Footnote 46} Not surprisingly, among OA journals, far more non-funded editors reported using a Gold OA model (73%) than continuously funded editors (12%). This difference is statistically significant (p<.001).^{Footnote 47} Conversely, significantly more continuously funded editors reported using a Delayed OA model (80%) compared to only 4% of non-funded journals.^{Footnote 48} ASJ administrative data suggest no increase in Gold OA among continuously funded journals since 2014, but conclusions are tentative as the analysis involves comparison of different datasets.

Figure 10 long description

Figure 10. OA model by age of journal

Age of journal in years	Frequency	Gold	Delayed	Hybrid	Other/None
More than 20 years	91	26%	36%	11%	27%
0 to 20 years	41	90%	5%	3%	3%

Influence of ASJ on OA strategy. Editors of journals who applied in 2018 were asked if the changes made to ASJ in that year affected their OA strategy. Slightly less than half of previously funded journals (48%, n=26) indicated that the changes did affect their OA strategy, with the same proportion (48%) responding that the changes did not affect their OA strategy.^{Footnote 49} Among journals funded for the first time in 2018, two-thirds (n=8) indicated that the changes did not affect their OA strategy.

Most editors reporting no change in OA strategy as a result of ASJ (80%, n=32) responded that their journal was already OA according to the 2018 requirements before the competition. All the responding continuously funded journals (n=10) reported already being OA or planning to be OA.

Among editors reporting a change in their strategy, a majority reported ASJ influencing the OA model chosen (Gold or Delayed) (65%, n=24). More than half (55%) also responded that the changes to ASJ sped up their OA transition.

Adoption of APC-based OA. APC-based OA has been reported to conflict with an equity and inclusion mandate because some authors have less access to APC funds, even when funds are made available as part of research grants. Public funding to OA journals has been proposed to support an equity mandate by enabling quality OA options at low or no cost to the researcher (see Question 5, below).

Researchers were asked in the survey if they had published in Gold OA format.^{Footnote 50} One in five were unsure, with the remaining 80% split evenly between those who had published in Gold OA and those who had not. The leading influences for those who had published in Gold OA were the availability of a free or low-cost option among their choice of journals or access to funding to pay author fees (APCs). Among researchers who had not published in this format (n=119), the most common reason was that the Gold OA option in their choice of journals required an author fee that was too high. This was reported by significantly more early career researchers (57%) than established researchers (27%).^{Footnote 51}

It is also important to note that in the researcher survey, OA options were reported to be of little or no importance for selection of a journal by nearly 31% of researchers, but essential or very important by a similar proportion (30%), indicating mixed demand for OA by SSH researchers. While slightly more early career researchers reported OA options to be essential or very important to them (41%) compared with mid-career (28%) and established researchers (23%), this difference is not statistically significant.

Journals’ use of APCs. Among editors responding to the survey, just over half (55%, n=72) reported providing some form of immediate OA, but only six (8%) reported that their journal charges APCs.^{Footnote 52} Only one of these six journals using APCs is currently ASJ-funded. Reported APC rates by these six survey respondents ranged from US$1894 to US$3400, with an average of US$2799.^{Footnote 53}

The evaluation also collected data from the websites of journals that were funded by ASJ through the period 2008-14, but which did not participate in the 2018 competition (n=22). Of these, 14 (64%) reported using a hybrid OA model and offered an immediate OA publication option.^{Footnote 54} All these journals reported charging APCs. Rates ranged from US$736 to $3060, with average of US$2,152.^{Footnote 55}

For comparison with international data, Crawford (2018) reports SSH OA journals to be 12% APC and 88% non-APC. More general estimates vary between 13% (Frantsvag & Stromme, 2019) to 47% (Morrison, 2018) charging APCs, with other estimates in between (e.g., Cantrell & Swanson, 2020). As noted above, of 59 OA journals currently funded by ASJ that responded to the survey, only one reported charging APCs (less than 2%). In terms of APC rates, inflation in these rates in recent years limits comparison with data from some older research studies: Papillon et al. (2019) report an average APC rate among SSH Gold OA journals at US$2880, similar to the rates reported in the survey for this evaluation; Crawford (2018) reports that only 5% of APC-based SSH journals charge more than US$600.

Participation in ASJ. Previously and currently funded editors were asked in the survey whether the 2018 changes to ASJ affected their decision to apply for funding. Of respondents that had been continuously funded (2008-14) but did not apply in 2018 (n=10), all reported that the new OA requirements were a disincentive. Among previously funded journals that participated in 2018, 30% (n=16) reported the additional funding for use of an OA platform had been an incentive to apply.

Twelve journals joined ASJ in 2018 for the first time (newly funded, with no prior applications). Among these, half (50%, n=6) reported the additional funding for use of an OA platform had been an incentive to apply, and 42% (n=5) reported that expansion of the eligibility requirements in 2018 had been an incentive to apply.

All but one of the journals new to ASJ (92%) are Gold OA and they now make up 10% of the ASJ-funded Gold OA journals. Close to half of the journals that had been continuously funded in the period 2008-17 did not apply to the 2018 competition (47%, n=22). Of these, 21 (95%) identified as subscription-only or hybrid journals.

Comparison with 2014 administrative data suggests that ASJ funding has increased in importance over time in terms of share of revenue of funded journals, although important limitations exist when comparing data from different datasets. Table 1 shows data for journals funded continuously by ASJ from 2008. In 2014, most reported ASJ to be less than 50% of revenue, while in 2020, most reported ASJ to be more than 50% of revenue. The shift may be due to the change in the composition of the funded journal pool as noted above, with some journals (with higher alternate revenue) leaving ASJ in 2018. It does not appear to relate to Gold OA transition of journals funded before 2018.

In sum. Information on journal response to the 2018 changes to ASJ was requested as feedback for program monitoring purposes. Over half of the journals funded continuously from 2008-17 did not participate in 2018. Most continuously funded journals that did participate in 2018 are OA journals (92%). Reported influence of ASJ’s 2018 changes on OA strategy is mixed, with about half of all participating editors reporting some impact on their strategy (48%), especially on model chosen and speed of transition, and about half reporting no impact.

Table 1: ASJ funding to other revenue, % journals by quartile (journals funded continuously from 2008)

ASJ to total revenue	% Journals in quartile
	2014	2020
	N=47	N=25
25% or less	19%	4%
26%‒50%	60%	31%
51%‒75%	15%	32%
76%‒100%	6%	32%

---

4.0 Cost-efficiency

4.1 Conclusions.

The data for ASJ indicate that ASJ is cost-efficient.

4.2 Detailed Findings on Cost-efficiency

ASJ’s cost efficiency ratios for the five fiscal years 2014-15 to 2018-19 are presented in Table 2, below. Over this period, the cost to administer the funding opportunity ranged from 2.40ȼ to 4.66ȼ for every C$1.00 granted, with an average administrative cost over the period of 3.23ȼ/C$1.00 granted. Accordingly, operating expenditures for ASJ ranged from 2.34% to 4.45% of total expenditures, with an average of 3.13% over the period. As 2018 was a competition year, this resulted in an increase in the operating ratios for ASJ in fiscal years 2017-18 and 2018-19.

The ASJ funding opportunity does not have a direct comparator within the SSHRC portfolio because it is a different type of funding model than other SSHRC programs. As a result, direct comparison of cost-efficiencies with another program is not informative. Instead, ASJ was considered relative to the average across SSHRC Research Grants and Partnerships, the category under which ASJ falls. The average for SSHRC programs in the Research Grants and Partnerships portfolio in 2018-19 was 3.84ȼ/$1.00.

ASJ’s average operating ratio of 3.23ȼ/$1.00 is below the 2018-19 average for Research Grants and Partnerships. Of note, ASJ’s relatively small size confers less economy of scale. Basic costs to run the program (e.g., to coordinate review and adjudication of applications) are spread over a small amount of grant funds relative to other SSHRC programs. Also, as noted above, ASJ costs are higher in competition years and lower in non-competition years. The period 2014-19 was unusual for ASJ for two reasons: (a) the standard three-year ASJ competition cycle was extended to four years^{Footnote 56} and therefore only one competition year is reflected in these data; (b) extensive stakeholder consultation and changes to ASJ were implemented in the latter part of this period, which may have affected administrative costs.

Table 2. ASJ program expenditures and efficiency ratios

	2014-15	2015-16	2016-17	2017-18	2018-19	Average
Operating expenditures	-	-	-	-	-	-
Direct salary	$27,274	$27,421	$27,558	$101,937	$110,464	$58,931
Direct non-salary*	$13,841	$0	$0	$0	$11,125	$4,993
Total direct	$41,115	$27,421	$27,558	$101,937	$121,589	$63,924
Indirect	$50,258	$52,326	$56,732	$52,801	$15,976	$45,619
Total operating expenditures (A)	$91,373	$79,747	$84,290	$154,738	$137,565	$109,543
Total grant expenditures (B)	$3,547,310	$3,329,055	$3,322,094	$3,322,094	$3,418,287	$3,387,768
Total program expenditures (C=A+B)	$3,638,683	$3,408,802	$3,406,384	$3,476,832	$3,555,852	$3,497,311
Operating ratio (¢:$1) Expenditures to grant funds awarded (A/B)	$0.0258	$0.0240	$0.0254	$0.0466	$0.0402	$0.0323
Operating expenditure as a percentage of total program expenditure (A/C)	2.51%	2.34%	2.47%	4.45%	3.87%	3.13%

Note: Includes Employee Benefits Plan (EPB)

---

5.0 Alternatives

5.1 Conclusions

In consultation with the Evaluation Advisory Committee, this question was directed at describing government interventions in other jurisdictions to support OA publishing, as reported in the peer-reviewed literature. A short summary follows. A full report on findings from this review was produced as a separate deliverable.^{Footnote 57} An important limitation is the small amount of research available on SSH. Most relies heavily on data about publications in health and natural sciences, for which practices differ.

Government mandates

The goals of funder mandates are typically to increase the overall amount of research published in some form of OA. Most policies specify deposit of the accepted or postprint manuscript (Gadd, 2018).

By 2018, over 50% of funders indexed in JULIET^{Footnote 58} either required or encouraged OA publication (Gadd, 2018). A minority of mandates prioritized Gold OA (estimates range from 2-25% of mandates). Green OA was more common^{Footnote 59} (see e.g., Pinfield, 2016) or no preference was stated in the policy (Gadd, 2018).

Estimates of researcher compliance with funder mandates vary but are generally low. Moreover, studies that rely on data about deposits in repositories may over-estimate compliance. This is because it is difficult to accurately exclude content that is not a research article (e.g., adjunct presentations and research summaries). Given apparent low compliance, authors recommend other incentives, training and supports (e.g., Borrego, 2016). Bakker et al. (2017) conclude that enforcement is necessary. However, other researchers caution that measures enacted in some jurisdictions to enforce mandates have resulted in substantial added administrative burden and cost on academic institutions. Compliance measures, if undertaken, need to be carefully designed (see e.g., Penn, 2018).

Government funding interventions

Four primary categories of government funding interventions to increase OA were identified. Advantages and disadvantages are documented for all (Table 3).

Table 3. Summary of OA interventions, reported advantages and disadvantages

	Description	Primary advantages	Primary disadvantages
Researcher-level funding	Usually direct support to authors / researchers to cover APCs. This funding is often distributed as part of the initial research grant.	Researcher choice. Potentially for greater uptake of non-predatory OA.	APC inflation. Quality and equity concerns.
Institution-level funding	Block grants or central payments to institutions to cover the cost of APC charges incurred by researchers at that institution, as well as other OA activities at the institution.	Reported increase in OA uptake. Simplicity for the funder. Institutional control over funding.	APC inflation. Lower non-APC OA uptake. Equity concerns. Administrative burden in institutions. Publishing sector consolidation.
Publisher or journal-level funding	Direct grants to OA publishers and journals are used particularly in countries with smaller populations than the US / UK, and where languages other than English are common.	Local, quality OA options with reduced or no APC. Funding leverage (e.g., to advance OA aligned with researcher interests).	Financial risk to journals / publishers of reliance on public funds, which may inhibit OA uptake. OA availability alone will not necessarily lead to researcher uptake of OA.
System-level funding	Support for the development of essential OA infrastructure (e.g., portals, repositories, OA software); also funding consortia although limited information.	Different types of support have different reported advantages. Example: consortia are potentially cost-neutral for libraries and revenue-neutral for journals in transition to OA. Consortia could work on other key tasks, e.g., quality standards.	Different types of support have different reported disadvantages. Example: consortia require minimum commitment from enough participants before journals likely to transition to OA, which is challenging to develop and sustain.

Primary overall themes in this body of work include the strong influence of researcher incentive systems on OA uptake. These include practices for assessing research publications in academic institutions. Attention to incentive systems and research assessment methods are therefore important for any strategy to support transition to OA. Support to efforts, such as the Declaration on Research Assessment (DORA)^{Footnote 60} to improve publication assessment, is one step, for example.

A second primary theme relates to unanticipated effects of funder interventions, which highlight the limited direct control of funders over outcomes and the dynamic nature of the environment. This suggests consultation and ongoing monitoring are necessary to mitigate adverse effects. The principles underlying the Tri-Agency Policy on Open Access (Appendix A) can provide a useful reference point to guide decisions where there is no known “best” solution and an adaptive strategy is needed.

A final theme is the importance of alignment among various funder policies in a jurisdiction and a need to support the development of common standards and best practices for discoverability.

---

6.0 Conclusions and Recommendations

6.1 Conclusions

This evaluation was designed to assess the relevance and performance of ASJ, with secondary questions on cost-efficiency and alternatives. Attention was paid to both intended and unintended outcomes and how they align with SSHRC’s commitments to support and advance SSH research and researchers in Canada. Relevance and alignment were a particular area of focus because ASJ is one of a small number of initiatives within SSHRC that do not fund individual researchers and research projects. These initiatives are aimed at shaping the environment to enable SSH research, or at underlying constraints in the SSH research system.

Relevance. As reported above, Canadian journals have served as a publication channel for a majority of Canadian SSH researchers (65%), and Canadian journals were rated as important channels for publication by 80% of those surveyed. SSH researchers reported these journals to be important, not only for disseminating their research, but in other ways as well, such as content for input to their research and to connect them with peer researchers and target audiences. Journals eligible for ASJ are operated primarily by volunteer academics on small budgets with few alternative sources of revenue.

The relationship between SSHRC and Canadian SSH journals is synergistic. SSHRC’s other investments in Canadian research and researchers are augmented by funding to journals that serve the same population. Journals are also positioned to serve researchers in a way that SSHRC is not. Taken from another perspective, the loss of capacity in Canadian publication channels would likely undermine SSHRC’s other investments, such as funding to research on topics of national importance.

Performance. ASJ funding enables a significant and meaningful increase in amount of SSH research published each year. ASJ makes small, indirect contributions in some other areas that relate to journal capacity to assure manuscript quality. To date, ASJ appears to have had small influence over accessibility and discoverability of the research published, but this influence has been located in key areas. The limited contribution of ASJ in areas beyond production capacity is not surprising given the small amount of funding available to journals through ASJ.

Cost-efficiency. The data for ASJ indicate that ASJ is cost-efficient. ASJ’s average cost-efficiency ratio is in line with the average for Research Grants and Partnerships overall.

Potential. ASJ could provide leverage for SSHRC on other commitments, such as to advance EDI. Canadian journals are positioned to influence EDI at the system level, and some journals already play a role to mobilize research on equity-related topics (e.g., Indigenous research). ASJ is also an important component of SSHRC’s effort to increase quality OA capacity in Canada. The funding is especially important to Gold OA journals for whom alternative revenue sources are very limited.^{Footnote 61}

6.2 Recommendations

1. Continue to offer ASJ funding. As long as the areas of relevance and performance described above are priorities for SSHRC, then continuing ASJ funding is a reasonable course. At this time, Canadian journals remain important to the SSH research system in Canada. ASJ is demonstrating positive outcomes that would be challenging to reproduce through other means. ASJ is a relatively small investment for SSHRC under a cost-efficient model.
   • In addition, alternate revenue sources for OA journals are currently limited or present equity issues, particularly for SSH researchers. Perceived risk of loss of public funding for OA, i.e., if programs are discontinued, is reported to inhibit OA adoption in the sector.
2. Adjust ASJ’s objectives to align with the Insight Program, considering ASJ’s key strengths and capacity. ASJ’s objectives are under-conceptualized relative to ASJ’s outcomes and potential within the Insight Program and how Canadian SSH journals are used and valued by researchers. Improved alignment with Insight Program objectives and the role of Canadian journals as identified by researchers (i.e., where these intersect) would improve ASJ’s focus and may enable it to do more with its limited resources.

   At a concrete level, this should include:

   1. Review of the Insight Program’s Program Information Profile to identify points of alignment between ASJ and Insight objectives and goals.
   2. Consideration for ASJ’s primary and potential outcomes given its niche and value as described by stakeholders, and its capacity, as documented above.
   3. Updating ASJ’s objectives to reflect how these intersect or have potential to align. Findings from the evaluation suggest that ASJ is contributing or could contribute to Insight Program objectives and goals. ASJ’s objectives could be more fully developed to better reflect and support this contribution. Examples are provided below to illustrate.
      • Research knowledge is produced and mobilized. ASJ’s current objectives focus mostly on dissemination. Dissemination is frequently described by researchers in terms of connecting their research with specific audiences―in other words, focused or targeted reach rather than broad dissemination. This is also reflected in ASJ adjudication criteria.^{Footnote 62} This has implications for performance indicators at the level of Insight, to ensure that these do not assume broad dissemination as the (only) desired outcome.
        • Research literature also identifies important benefits of domestic journals in terms of service to groups not well served by other channels (e.g., non-English language researchers) and the development of quality, equitable OA channels.
      • Enabling individuals and teams to engage in research and research-related activities (including collaborations within Canada). While this is not yet reflected in ASJ’s objectives, researchers describe the value of Canadian journals to their work in ways that include connecting with other researchers working in their field and for source material to advance their research. Research literature also identifies cultivation of local researcher communities in specific fields as an important benefit of domestic journals.
      • Strengthening EDI in SSH research. ASJ’s objectives do not currently incorporate EDI. However, academic journals, editors and peer reviewers play an important role in EDI at the system level and opportunities for ASJ to contribute at this level are reported above. If ASJ’s potential contribution to Insight’s EDI goals were to be recognized in ASJ’s objectives, this would need to reflect the organizational-level nature of the funding and the system-level nature of the intervention. There is substantial diversity among Canadian SSH journals and varying levels of capacity. There are also many possible ways in which journals might engage with EDI. Soliciting expert advice would be suggested. In the meantime, EDI monitoring is recommended (see below).
      • Supporting the next generation of social sciences and humanities scholars. ECRs were more likely to rate Canadian journals as essential or very important for connecting with others in their field and to rate Canadian journals as essential or very important to career advancement.
      • Enabling research trainees to enhance their research, professional and knowledge mobilization skills. Funded journals often employ and train graduate students.
3. Ensure operational alignment of ASJ to updated objectives. More fully developed objectives should be carried forward into the ASJ review process and performance monitoring practices. For example, ASJ adjudication criteria should be reviewed to ensure alignment with, and appropriate emphasis on, core priorities. For example, if contribution to the quality of published research is a priority objective for ASJ, the scoring rubric and process should reflect this priority. The Evaluation Division could provide technical assistance to develop an analytical rubric, e.g., with distinctions between performance levels to aid reviewers.
   1. Adjudication criteria should be reviewed against EDI commitments to ensure, at minimum, that barriers to funding are not unintentionally introduced based on characteristics associated with the journal, such as, for example, composition of the editorial board, regional scope or topic focus. As an example, a journal that specializes in a younger field of research or interdisciplinary research may have a very different citation profile than a journal in a larger, more established discipline.
   2. Taking the first steps to collection of data related to EDI is also recommended, with a focus on monitoring for indicators related to equitable access to ASJ funding. As ASJ funds at the organizational level, expert advice is advisable to identify appropriate and feasible options for data collection aimed at organizational- or system-level.
      • ASJ may wish to consider data that could support program management to ensure that the ASJ program structure or delivery does not unintentionally create barriers to funding for a journal.
   3. Adjudication criteria and performance indicators should also be reviewed against the recommendations of the DORA declaration^{Footnote 63}―to which SSHRC is a signatory―as a further way to support conditions for equitable participation in the research system.
4. Update ASJ’s logic model. ASJ’s current logic model (produced at the start of this evaluation) does not accurately capture ASJ’s performance nor the mechanisms by which it contributes to outcomes. In other words, how ASJ works is not explicit in the existing model. The findings reported above provide the basis for an empirically based change model to guide ASJ and/or inform similarly structured funding opportunities. The Evaluation Division can provide technical assistance.
5. At the corporate level, consider using ASJ as a vehicle to further advance key council priorities.
   • The current investment in ASJ is not sufficient to produce more than a very modest contribution to the amount, quality and accessibility/discoverability of published Canadian research. If SSHRC wants to do more to advance these outcomes, SSHRC could raise the grant amount. Additional funds would likely be applied by journals to production capacity, including peer review coordination, with some spillover effect in other areas such as strategic development. It could also be targeted to areas identified for attention as reported above, such as in some discoverability practices.
   • ASJ can contribute to SSHRC’s learning as an organization in key priority areas through close engagement with the sector and by keeping communication lines open. This is especially applicable for areas that are developing rapidly, such as:
     • Capacity for action at a system level on EDI, given that some SSHRC initiatives, like ASJ, are positioned at a system level;
     • Ongoing efforts to improve research assessment by academic institutions and by SSHRC adjudication committees, i.e., DORA implementation, so that researcher incentive systems do not inhibit OA publication, for example, French-language publication, or publication choices that prioritize reach to key target audiences over broad dissemination when this is important for research use;
     • Efforts to advance understanding of “open” research sector-wide and the recognition of OA model adoption to achieve this larger objective. This includes other practices important to accessibility/discoverability but not yet widely adopted in the sector, as reported above. Some options include sharing quality information about developments in open research, supporting development of “best” practices and policies for open research among SSH journals, support for accessible/low-cost technology and access to intermediary expertise. This is also consistent with DORA recommendations (above), for example with respect to reuse rights for citation data;
     • Exploring options for the development of equitable and sustainable OA resourcing.

---

References

Bakker, C., Stephenson, C., Stephenson, E., & Chaves, D. (2017). Public funding and open access to research: a review of Canadian multiple sclerosis research. Journal of Medical Internet Research, 19(2), e52.

Björk, B. C. (2017a). Journal portals–an important infrastructure for non-commercial scholarly open access publishing. Online Information Review, 41(5), 643-654.

Björk, B. C. (2017b). Scholarly journal publishing in transition-from restricted to open access. Electronic Markets, 27(2), 101-109.

Borrego, Á. (2016). Measuring compliance with a Spanish Government open access mandate. Journal of the Association for Information Science and Technology, 67(4), 757-764.

Bower, K., Sheppard, N., Bayjoo, J., & Pease, A. (2017). Establishing the role and impact of academic librarians in supporting open research: a case study at Leeds Beckett University, UK. New Review of Academic Librarianship, 23(2-3), 233-244.

Crawford, W. (2018). GOAJ3: Gold Open Access Journals 2012-2017. Livermore, CA: Cites & Insights.

Eng, S. (2017). The library profession in the time of open access. The Serials Librarian, 73(3-4), 215-225.

Gadd, E., Fry, J., & Creaser, C. (2018). The influence of journal publisher characteristics on open access policy trends. Scientometrics, 115(3), 1371-1393.

Gadd, E., & Troll Covey, D. (2019). What does ‘green’ open access mean? Tracking twelve years of changes to journal publisher self-archiving policies. Journal of Librarianship and Information Science, 51(1), 106-122.

Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29-29

Lohr, S. (2019). Sampling: Design and Analysis. Boca Raton, FL: Taylor & Francis.

Penn, L. (2018). Alternative ways of obtaining scholarly articles and the impact on traditional publishing models from a UK/European perspective. Serials review, 44(1), 40-50.

Pinfield, S., & Middleton, C. (2016). Researchers’ adoption of an institutional central fund for open-access article-processing charges: A case study using innovation diffusion theory. SAGE Open, 6(1), 1-18.

Piwowar, H., Priem, J., Larivière, V., Alperin, J. P., Matthias, L., Norlander, B., ... & Haustein, S. (2018). The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ, 6, e4375.

Ryan, K., Gannon-Slater, N., & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small-and medium-scale evaluations. American Journal of Evaluation, 33(3), 414-430.

Statistics Canada (2015). Personnel engaged in research and development by performing sector and occupational category, Table 27-10-0022-01.

Willinsky, J. (2017). Modelling a cooperative approach to open access scholarly publishing: A Demonstration in the Canadian Context. Canadian Journal of Communication, 42(5).

Zhang, L., & Watson, E. M. (2017). Measuring the impact of gold and green open access. The Journal of Academic Librarianship, 43(4), 337-345.

Zhang, L., & Watson, E. (2018). The prevalence of green and grey open access: where do physical science researchers archive their publications? Scientometrics, 117, 2021-2035.

---

Appendix A—ASJ Program Profile

Overview

ASJ’s overall objective is to increase dissemination of, access to, and readership for original Canadian research results in the social sciences and humanities (SSH).

Before 2018, the program sought to help defray the costs of publishing scholarly articles, to assist with distribution costs, and to support journal organizations in transitioning to and maintaining digital formats as a necessary pre-condition to moving toward open access. In 2014, journals were asked to submit a plan describing how they would transition to an open access model.

Changes brought about in 2018 have shifted the purpose of the grants somewhat. In addition to increasing dissemination, discoverability and readership of original SSH research results, the program now seeks to leverage significant federal investments in digital scholarly publishing infrastructure; to assist journals to take advantage of developments in digital technologies and innovative practices; and encourage the transition to open access models of publishing.

Before 2018, the ASJ funding opportunity allowed journals to seek support regardless of business model or distribution format. Changes to the program in 2018 now require a journal to be able to offer either immediate open access upon publication or delayed open access with a 12-month maximum embargo period. Journals not compliant with an eligible OA model were allowed two years to make the transition. The third year of funding is contingent upon a successful transition having been made.

ASJ’s intermediate objectives are as follows:

• increase dissemination, discoverability and readership of original research results in the social sciences and humanities through Canadian scholarly journals;
• leverage significant federal investments in digital scholarly publishing infrastructure;
• assist journals as they seek to take advantage of developments in digital technologies and innovative practices; and
• encourage the transition to open access models of publishing

Four ASJ competitions have been held over the past ten years: in 2008, 2011, 2014 and 2018.

Table 4: ASJ Competition Statistics

	2008	2011	2014	2018	Total
# Applications	186	193	197	175	751
# Awards	162	138	144	123	574
Success Rates**	87%	72%	73%	70%	-
Total Amount Awarded	9,963,409	9,999,942	10,198,459	11,600,007	30,161,810
Average Amount Awarded	61,503	72,463	70,823	79,089	-

**Success rates are calculated based upon the number of awards and the number of applications. They may change once withdrawn and ineligible applications are factored into the success rate calculation.

---

ASJ expenditures in this period have remained quite stable, at between $3,188,156 to $3,547,310 per fiscal year, with higher expenditures in the years that competitions were launched.

Key changes to requirements for ASJ funding in 2018 included:^{Footnote 64}

• A requirement that journals be open access (OA) immediately upon publication or have an embargo period of no greater than 12 months;
• A transition period of two years to comply with OA requirement;
• New eligibility for journals that can demonstrate that their titles are owned by Canadian not-for-profit organizations and have a Canadian affiliated editor-in-chief but do not meet the requirement for 1/3 of members to be affiliated with a Canadian postsecondary institution

Application Process, Grant Administration and Adjudication

Applications must be submitted by the editor-in-chief of the journal. Grant funds may only be administered by an eligible postsecondary institution or eligible not-for-profit organization (which, for ASJ, is defined as either the journal itself or a scholarly association or press that either administers or owns the journal). Institutions or organizations proposing to administer any grant awarded under this funding opportunity must hold or obtain institutional eligibility. Institutional eligibility^{Footnote 65} of a journal, scholarly association or press applies only to the ASJ funding opportunity and must be renewed for each competition.

All applications undergo adjudication by scholars from a variety of disciplines who are well versed in journal operations.

The following criteria and scoring scheme are used by adjudication committee members to evaluate ASJ grant applications.

1. Quality and impact of the journal (40%):
   • Rigour and quality of the processes used to select and peer review articles;
   • Relevance of the articles published in relation to the journal’s objectives;
   • Distinctiveness of the journal’s content;
   • Value of the contribution the journal has made to its field(s) of research; and
   • Impact of the journal’s articles (e.g., as measured through citation indexes, traditional metrics or altmetrics, or as described by the applicant).
2. Editorial direction (30%):
   • Leadership provided by the editor-in-chief;
   • Appropriateness of the editorial board membership/publishing team membership;
   • Evidence of the active engagement of board/team members; and
   • Soundness of the journal’s proposed objectives for the period covered by the grant.
3. Nature and effectiveness of the journal’s current and proposed dissemination strategies (30%):
   • Diversity and relevance of the journal’s proposed strategies for reaching the appropriate readership;
   • Nature and extent of the journal’s readership, and of the journal’s efforts to increase or broaden that readership;
   • Where applicable, viable plans for transitioning to an open access business model;
   • Overall appearance and presentation of the journal’s content; and
   • Design and usability of the journal’s website.

A score is assigned on a scale of 1-6 for each of the criterion above. Applications receiving a score of 3.0 (satisfactory) or higher in each of the above criterion may be recommended for funding. Each criterion score is weighted by the values listed above to arrive at a final score.

Tri-Agency Open Access Policy

The Tri-Agency Open Access Policy, as described in detail below, is relevant to this evaluation. The objective of the policy is to improve access to the results of agency-funded research and to increase the dissemination and exchange of research results. All researchers, regardless of funding support, are encouraged to adhere to this policy.^{Footnote 66}

The following principles guide the agencies in their approach to promoting open access to research publications:

• Committing to academic freedom and the right to publish;
• Recognizing the critical importance of peer review to the scholarly communication ecosystem;
• Maintaining the high standards and quality of research by committing to academic openness and responsible conduct of research;
• Promoting recognized research best practices and standards across disciplines, and embracing and sharing emerging practices and standards;
• Advancing academic research, science and innovation;
• Effective dissemination of research results; and
• Aligning activities and policies between Canadian and international research funding agencies.

---

Appendix B—Methodology

As this was a joint evaluation for ASJ and ASP, a common approach was used for both initiatives. Common methods were also used wherever possible. Specifics on data collection and analysis methods are provided below. Methods applied only to ASP and not ASJ are not included below but are documented in the evaluation report for ASP.

Key informant interviews

Key informant interviews were conducted as one line of evidence for the evaluation. The purpose of the interviews was to gather exploratory data in Phase 1 relevant to evaluation questions and indicators, and as part of developing a draft theory of change. The results were also used to inform development of Phase 2 instruments.

Interview participants were selected by the SSHRC Evaluation Division drawing on administrative data for ASJ and ASP to provide a broad cross-section of perspectives. Efforts were made to ensure the sample within each subgroup included representation from across regions of Canada (i.e. West, Central, and Atlantic) and official languages (i.e. French, English). A total of 42 interviews were conducted across eight sub-groups, as follows:

• Editors of journals funded by ASJ (n=8)
• Editors of journals not funded at all or funded first by ASJ in 2018 (n=4)
• Publishers that are frequently funded by ASP (n=7)
• Publishers that are rarely funded by ASP (n=4)
• Academic researchers in the social sciences and humanities (n=6)
• Other stakeholders and representatives of stakeholder organizations (n=6)
• Program representatives (ASJ, ASP) (n=5)
• Government representatives (other federal funders) (n=2)

SSHRC’s Evaluation Division developed semi-structured interview guides for each sub-group in consultation with Ference & Company Consulting and the EAC. Interview guides were tested by Ference & Company. Ference & Company invited the participants, conducted the interviews and analyzed the data. Interviews were conducted by telephone in the official language of the participant’s choice. Participants were provided a copy of the relevant interview guide before the interview. Interview notes and audio recordings were used to compile transcripts of each interview.

An initial codeframe for analysis was developed by Ference & Company in consultation with SSHRC. Transcribed responses to each question were entered into a MS Excel spread sheet for coding and analysis by Ference & Company.

ASJ Retrospective Cohort Study

This study assessed the effect of ASJ funding on the quantity of published articles among Canadian journals between 2008 and 2017.

The following eligibility criteria were used:

• Meet the definition of a “scholarly journal;”
• Be either a subscription-based journal, a journal using an open access business model or a journal using a “moving wall” arrangement to provide delayed open access;
• Submit each article to be published to a rigorous, independent peer-review process;
• Have been founded at least two years before the application deadline and have published, within that timeframe, a minimum of two issues per year, each having at least three peer-reviewed articles, or, for digital journals not publishing in issue format, at least 12 peer-reviewed articles per year over the two-year timeframe;
• Publish primarily in English and/or French;
• Conform with SSHRC’s Subject Matter Eligibility guidelines.

A journal is ineligible for ASJ funding if:

• Its mandate and/or content is primarily focused outside of discipline(s) within SSHRC’s mandate; and/or,
• Its content is primarily health-related or aimed at improving, promoting or influencing health, health care, health policy, or healthcare-related services.

A journal that received funding between 2008 and 2017 was included in the study if it was also included in the Web of Science database. A journal that did not receive funding from the ASJ funding opportunity was included as the comparator if it was included in the Web of Science database and met the most relevant eligibility requirements for funding. Eligibility for the non-funded group was assessed using data between 2005 and 2017. ASJ-funded journals had additional eligibility requirements for the editorial board; however, composition of the editorial board was not determined for the non-funded group due to lack of available data.

The primary outcome of this study was the average number of annual research article publications per journal. The main factor of interest was funding through the ASJ funding opportunity. This factor was further assessed as being fully funded during the entire duration of study (2008 through to 2017) or receiving partial funding throughout the study period. Journals that did not receive any funding from ASJ between 2008 and 2017 were the comparator group. Journals that received funding from ASJ between 2005 and 2007 but no funding from 2008 to 2017 were excluded to remove the potential carryover effects of funding.

The primary language of publication, the journal discipline and the frequency of publication were included as potential confounding variables. Data for the primary outcome were gathered from the Web of Science. The number of articles published was extracted for each journal. The sum of publications was divided by the number of years of data that was available for each journal to produce an annual average publication rate. The annual publication rates were positively skewed and followed a Poisson distribution. As such, the publication rate was adjusted using a logarithmic transformation to approximate a Normal distribution. All analyses were conducted on the annual log publication rate.

The main exposure of interest was determined from the ASJ administrative data file. After all journals were selected, they were cross-referenced with the administrative data file to determine whether they were a funded or not-funded journal.

Journals could not be randomly selected for inclusion as measurement of the outcome was dependent on data being available in the Web of Science database. To address potential selection bias, journals that were funded and included in the study were compared to journals that were funded but not included in the study to determine if there were any important differences in baseline characteristics.

All journals that met the eligibility criteria and had data available in the Web of Science database were included. Publication counts were converted into annual publication rates. This allowed a single number to represent the outcome for each journal when not all journals were included in the study for the same amount of time. This prevented journals that had 10 years of data from having higher publication counts simply because more data were available. As not all journals were funded for the entire duration that data were available, funding status was stratified into two categories: full funding and intermittent funding.

Every article was assessed for publication language. The total number of English and French articles were summed over the period of inclusion in the study for each journal and divided by the sum of articles to determine the proportion of the total that were published in each of these languages. Journals that published more than 50% of their articles in English over the study period were identified as English majority and journals that published more than 50% of their articles in French over the study period were identified as French majority. Therefore, language of publication was included in the analysis as a binary variable.

The primary analysis that assessed for the effect of funding on quantity used multiple log linear regression controlling for language, discipline and frequency of publication. Subgroup analysis looked at the effect of full funding and intermittent funding. No interactions were included in the final model. There were no missing data.

The analysis was run twice: in the first analysis, the regroupings were conducted by an evaluator and eligibility criteria of the non-funded group were less constrained; in the second analysis, a second evaluator independently regrouped the variables and the eligibility criteria were more closely matched to those of the ASJ program. In both analyses, the results were similar, indicating that the effects seen were likely not due to bias being introduced in the regrouping of variables.

All analyses were conducted using IBM SPSS version 23.

ASJ Administrative Data Analysis (Qualitative Component)

This component was designed to identify themes across multiple grantee perspectives and to consider how they may vary over time. A stratified random sample was drawn from successful applications to ASJ. These applications were submitted by editors of Canadian SSH scholarly journals: 15 applications were drawn from each of the four competitions in the 2008-18 period for a total of 60 applications. To ensure a broad cross-section of perspectives in the sample, representation was sought from disciplinary categories, geographic regions, and francophone and anglophone applicants.

Data from the 2011, 2014 and 2018 applications were coded in NVivo^{Footnote 67} following a pre-defined coding framework, then open-coded within categories to identify primary subthemes. These subthemes were then reviewed for internal coherence, with reference back to the original data as needed. Data were recoded where appropriate to respect the emergent coding structure.

The 2008 applications were then reviewed for correspondence and/or divergence with the final subthemes and higher-order categories. Data from these applications were integrated manually into a results summary in MS Word. The final content of each subtheme was then reviewed by competition year to assess possible change over time.

This analysis relies on self-reported data in funding applications and is assumed to carry a positive bias with respect to anticipated and reported outcomes. Accounts may be influenced by participant understanding of SSHRC organizational objectives. For this reason, the analysis was not intended to draw conclusions about the achievement of outcomes, and instead focused on reported processes and mediating factors. There was no opportunity to seek clarification from applicants. Effort was made to avoid inferring connections between factors, i.e., between ASJ funding and particular outcomes; instead the analysis only considered connections that were explicitly articulated by the applicant.

ASJ Administrative Data Analysis (Quantitative Components)

Data on journal submissions and access models. Data regarding domestic (Canadian) submissions were compared for continuously funded applicants, intermittently funded applicants (i.e., unsuccessful in the competition year but successful in other years), and never-funded applicants (i.e., unsuccessful in all years in the period) from 2008-14 ASJ cycles. Detailed submissions data were not available beyond 2014.

The submissions data are self-report data for the two years before the competition year, drawn from ASJ applications. A sample of 32 applications was drawn from each cycle (2008, 2011, 2014, and 2018) for a total of 128 applications. Data were then manually assessed and captured from these applications for analyses.

Journal revenue data. Data were collected from ASJ applications of journals that had received ASJ funding continuously from 2008-14 (continuously funded journals; n = 53). Continuously funded journals are likely to be more well-established and financially stable than other journals.

Journals reported their financial data in ASJ applications up to and including the 2014 competition. In the 2018 competition, these data were no longer required on applications. Six journals had non-usable data and were excluded from the analysis, resulting in a total of 47 journals in the dataset.

The following data were extracted from the 2014 ASJ funding applications of these 47 journals:

• journal total revenue for the 2013-14 year;
• the amount of the ASJ grant they received in 2013-14; and,
• journal business model category.

The ASJ grant received was divided by the total revenue for the journal to obtain the proportion of funding that ASJ represents for each of the 47 journals.

Journals were then grouped by quartile as follows: journals for which ASJ funding represents 0-25% of total revenue, 26-50%, 51-75%, and 76-100%.

These results were further developed via survey (below).

ASJ cost-efficiency. SSHRC typically assesses cost efficiency by calculating the ratio between a program’s operating costs and grant funds awarded (the operating ratio). Data for this analysis were supplied by staff in the office of Awards Management, Compliance and Accountability. The data included SSHRC operating, grant and total expenditures for ASJ from fiscal year 2013-14 to 2018-19.

SSHRC CCV Data Analysis—Canadian SSH Researcher Publications

A cross-sectional design was used to study researcher publication histories in order to produce estimates of the proportion of the Canadian SSH research population that had published with Canadian journals and book publishers.

This study used data extracted from the SSHRC Canadian Common CV (CCV) database on December 2, 2019. The CCV data were compared to the SSHRC AMIS dataset on researcher demographic variables and SSHRC program participation^{Footnote 68} to ensure representativeness of the CCV data to the larger AMIS dataset, that, in turn, is considered representative the Canadian SSH research population (see below). For comparison with SSHRC program participation, the records of individuals above 90 years of age or marked as deceased were excluded.

The target population was all Canadian social sciences and humanities (SSH) researchers, defined as academic researchers working in SSH and affiliated with Canadian institutions. The study population was drawn from the SSHRC CCV and consisted of a sample of Canadian researchers who have created or updated their CCV between January 1, 2017 and December 2, 2019. Researchers who were not currently affiliated with a Canadian institution or who had no publication data available were excluded from this analysis. The sample was selected using simple random sampling to draw a study sample that would be representative of the total study population. The size was calculated to produce estimates of proportion published through Canadian publishing channels and also based on the needs of compiling a sample frame large enough for a planned survey of researchers for the evaluation, with as small a margin of error as possible within the constraints of our resources. It is estimated that there are approximately 28,000 SSH researchers in higher education in Canada (excluding graduate students).^{Footnote 69} Our study population was large enough to provide coverage for 16% of this population.

Data from the SSHRC CCV were extracted using SAP BusinessObjects Web Intelligence and exported into a CSV file. Data management and cleaning was done in Microsoft Excel 2016. Descriptive statistics were computed using IBM SPSS version 26. Researchers with missing values for both journal and book publisher were excluded from the analysis. No imputation was performed.

The entire population of researchers contained within the AMIS database was extracted and used as a proxy of the Canadian population of SSH researchers. AMIS records are considered a reasonable proxy because of the overall number of unique records (36,874 relevant records were identified) reflecting researchers who have had contact with SSHRC over an extended period. A unique identifier was extracted from AMIS for each individual, along with year of birth, gender, language, institution, institution type, PhD completion date and PhD discipline. These variables were attached to the researchers in the CCV dataset using the unique identifier. Of the 4,631 records in the CCV population, 257 remained without a match on the unique identifier to AMIS. These records were further assessed against AMIS records based on name, institutional affiliation and discipline to confirm a match. A match was ambiguous or not found for 98 records. These records were not included in the subsequent assessment. The final sample totaled 1,306 researchers currently affiliated with a Canadian institution and with a publication list.

Limitations: Assessment of SSHRC CCV data indicates that it is representative of the total SSH researcher population in Canada by geographical location and institutional proportion. However, about 66% of researchers using the SSHRC CCV are emerging researchers. Small but significant differences in mean age between the two datasets were identified, with a younger mean age in the CCV dataset than the AMIS population. In addition, women and French-language researchers have slightly higher representation in the CCV data, and the individuals in the CCV have small but significantly greater participation rates in most of SSHRC’s core funding opportunities, indicating slight but significantly greater engagement with SSHRC across SSHRC programs compared with the AMIS population. In addition, during the data cleaning phase of this analysis, it was discovered that some of the researchers in the SSHRC CCV sample started out as a SSH researcher, but may have shifted toward a STEM discipline as their career progressed. As the total publication history was assessed regardless of discipline, there may be some misclassification of researchers, which could bias the results. There was no attempt to control for these biases in the sample. However, respondent disciplinary affiliations in SSH and representation by stage of career were later confirmed through the survey.

Given skew to younger researchers, a weighted analysis of the CCV data was conducted to compensate for bias in age, with the result producing a slightly higher estimate of proportion published through a Canadian channel than unweighted analysis. The weighted analysis is reported in the evaluation findings above.

Document and Literature Reviews

Five focused reviews of documents and peer-reviewed literature were conducted over the course of the evaluation. Three reviews took place in Phase 1 to identify and summarize relevant findings from previous studies related to ASJ: these were used as a supplementary line of evidence on their relevance and performance. These reviews also identified options for evaluation indicators for Phase 2. Two reviews were focused on question 5 of the evaluation related to alternative approaches. The fourth review was conducted in Phase 2, focused on EDI.

1. Relevance review: scholarly literature and funding program documentation related to funding to scholarly publishing in Canada (n=22).
2. Performance review: past evaluations and studies specific to ASJ and ASP (n=7).
3. Alternatives review: peer reviewed literature published 2016-May 2019 on government interventions to advance OA publishing (n=69).
4. Review of literature on the use of article-processing charges (APCs) for OA publication in SSH journals, international benchmarks (n=7).
5. EDI review: SSHRC and Tri-agency documents on EDI and ASJ/ASP program documentation related to objectives and program collection of monitoring data (n=8).

Literature review (Government funding approaches for OA)

This review focused on articles in peer-reviewed journals published between January 2016 and May 2019^{Footnote 70} that describe funding approaches used by government agencies that have explicit objectives to support open access research dissemination. For relevance and feasibility, the review was scoped to include only government funding initiatives. It did not attempt to cover policy initiatives.

The review was intended to provide descriptive information for use by program stakeholders on the following questions. These questions guided the review:

As reported in recent peer-reviewed journal articles:

1. What are the most prominent funding approaches used by other government agencies to promote open access research dissemination?
2. Why have these approaches been chosen (i.e., primary reported rationale, if given)?
3. What are the reported advantages and/or disadvantages associated with each approach?

The literature search was undertaken in four steps: Title review (n=286), Abstract review (n=90), Full text review (n=80), Additions from reference lists of included articles (n=3). A final dataset of 69 articles was included in the review.

Synthesis

Relevant data were extracted from the articles following a pre-set coding framework in NVivo. The coded data were then condensed into a data display matrix to isolate information related to the guiding questions. A narrative descriptive synthesis of the results was developed from the matrix.

Limitations

This review did not assess the included studies for methodological quality and it did not conduct an analytical comparison of content (for example, between international and Canadian sources). Not all the articles in this review are based on empirical research; some are commentaries. The review was not designed to include a critical analysis, interpretation or evaluative judgement of article content, and does not present recommendations.

Much of the peer-reviewed literature on the topic of OA and government intervention focuses on natural and health sciences and does not discuss social sciences or humanities. Items that focused exclusively on natural or health sciences publishing were excluded unless they also mentioned other fields and/or explicitly referenced policy or funding programs in Canada. This was to ensure the relevance of the results to SSHRC and the SSH domain. Most of the included literature focuses on countries other than Canada. Applicability to the Canadian context is tentative.

Surveys

Three surveys were conducted in Phase 2 of the ASJ-ASPP evaluation, one for each of three populations:

1. Canadian SSH journal editors;
2. Canadian scholarly publishers funded by ASPP; and
3. Canadian SSH researchers who have published through a Canadian scholarly journal or book publisher.

The purpose of these surveys was to complement and build on Phase 1 results by addressing gaps in Phase 1 data and assessing whether factors identified in Phase 1 qualitative analysis can be generalized across subgroups and populations of interest. The journal editor survey applied to ASJ only. The publisher survey applied only to ASP. The researcher survey applied to both funding opportunities.

A total of 132 editors fully completed the survey for ASJ, which accounts for 46% of the total estimated population (N=286). The margin of error (95%) is 6%, which is generally considered appropriate for survey data.

314 researchers fully completed the researcher survey, which is 32% of the population sample (n=991). The margin of error (95%) is 5%, which is also appropriate for survey data: this indicates that if the results were generalized to the population, the range of values above and below the survey value should fall within 5%. However, as noted above, both populations are heterogeneous. The results may be less generalizable to some subgroups within the population.

More information on the editor and researcher surveys are provided below.

Editor survey

The editor survey was distributed to lead editors of Canadian SSH scholarly journals for whom valid contact information could be identified and whose journals met basic requirements from ASJ eligibility criteria (N=286).

The survey instrument was developed through multiple rounds and was twice live piloted via interviews to improve response process validity. A revised draft of the survey instrument was reviewed by the Evaluation Advisory Committee. The instrument was further adjusted based on feedback from the EAC review. The survey was translated into French and programmed for online use in Interceptum, a Canadian survey hosting platform, in January 2020. The survey was launched on January 24, 2020 and remained open until February 25, 2020.

Table 5. ASJ Editor Survey Completion Rate and Margin of Error

ASJ Editor Survey
Population	286
Number of editors disqualified for indicating that they are not now, nor have they recently been, an editor of a Canadian scholarly journal	2
Number of surveys abandoned	21
Completed surveys	132
% Complete	46%
Margin of error (95%)	6%

---

Researcher Survey

The population was defined as SSH researchers affiliated with Canadian institutions, who have used Canadian journals or book publishers as a publication channel. This population definition was used to enable a response sample large enough for subgroup analysis of potential differences in the use of Canadian publication channels (e.g., by career stage, language or disciplinary affiliation).

To respect the limited scope of this evaluation, the researcher sample was drawn from two available administrative data sources:

1. Canadian SSH researchers with Canadian publication records in the SSHRC Canadian Common CV (CCV) database.

   To estimate an appropriate sample size for the survey, a pilot study using the SSHRC CCV data was completed. Based on the results, a sample of 1,500 was drawn from the CCV data using simple random sampling to be representative of the total study population. Individuals with no Canadian affiliation and no publication history were removed, along with researchers who had not published in a Canadian journal or through a Canadian book publisher. This left a sample of 865 researchers.

2. ASP applicants in the years 2016 and 2017. These were included in the sample to ensure sufficient representation of researchers published with Canadian scholarly book publishers. These application years were chosen to allow time for submitted books (successful and not successful) to have been published. Both successful and not successful applicants were included. Applicant data were provided by the FHSS from the ASPP administrative database (n=522). 91 Individuals not affiliated with a Canadian institution were removed from the list, and a random sample of 50% was then extracted (n=215). 27 records were excluded due to duplication or because a valid email address could not be identified (n=188).
   

The ASP and CCV lists (n=1053) were compared to identify duplicates before merging the lists. The list was also compared to the list for the ASJ editor survey to ensure researchers would not be surveyed twice: 24 duplicates were removed (n=1029).

The initial invitation to the sample generated 36 automated replies from individuals on leave (e.g., sabbatical, parental, medical, unspecified). One additional individual was removed because a valid email address could not be found. The final sample of 993 was then transferred to Ference & Company for survey administration.

The survey instrument was developed collaboratively between Ference & Company and the SSHRC Evaluation Division, with input from the EAC, between November 2019 and January 2020. As the Evaluation Division and Evaluation Advisory Committee contain members who are researchers with publications in Canadian channels, live pilots with external researchers were not conducted. The survey was translated into French and programmed for online use in the survey software Interceptum. Online testing of the survey in English and French was conducted internally over multiple rounds at the end of January. An online pilot in English was conducted in January with a Canadian SSH researcher. In early February, the French version of the survey was piloted by a Francophone Canadian SSH researcher. This pilot generated no further changes.

The survey was soft-launched on February 14, 2020. The survey was fully launched on February 18, 2020. A reminder email was sent to all recipients who had not already completed the survey on February 24, March 2 and March 6, 2020. The survey closed on March 10, 2020.

Approximately one-third (32%) of researchers in the population sample fully completed the survey. Four researchers responded but were disqualified and 23 did not complete the entire survey; their responses are not included in any analyses. The margin of error of 5% indicates that when generalizing the results of the survey to the population, the range of values above and below the survey value is 5%.

Table 6: ASJ-ASPP Researcher Survey Completion Rate and Margin of Error

ASJ-ASPP Researcher Survey
Population (Sample)	28 000 (991)
Number of researchers disqualified for indicating that they are not a researcher in SSH or affiliated with a Canadian institution	4
Number of surveys abandoned	23
Completed surveys	314
% Complete	32%
Margin of error (95%)	5%

---

Survey Analysis

Closed-ended responses

Survey data were extracted from the survey platform to MS Excel format. Data were cleaned and verified for analysis in SPSS. Analysis of the closed-ended responses was conducted by Ference & Company.

Significance testing:

Significance testing for variations in survey responses in the researcher survey was undertaken using appropriate tests, including independent-samples t-test, one-way analysis of variance (ANOVA), and chi-square test of independence in SPSS.

The editor survey was conducted with a small, finite population, and all members were included in the frame. Where subgroup sizes fell below the minimum, techniques specific to small samples or finite populations were applied (e.g., Fisher’s exact test or incorporating the finite population correction factor in the journal editor survey).^{Footnote 71}

Open-ended responses:

Open comment responses were extracted from the survey platform to MS Excel format. Respondent attributes (e.g., unique ID, subgroup membership status) were also extracted. Survey data were then imported to NVivo for analysis by SSHRC’s Evaluation Division.

The data were coded inductively into categories by evaluation question. Once all records had been coded, the contents in the categories were reviewed for internal consistency, distinctiveness from other categories, and to confirm the representativeness of the category label to content held in the category node. Where necessary, content was disaggregated into subcategories to improve internal consistency, categories were merged when insufficiently distinct or underpopulated, and category labels were modified to better reflect the final content set. Where a comparison by subgroup was relevant, a crosstab query was used to obtain reference and respondent counts (frequencies) by subgroup for each node category.