Trends

We chose a set of journals that many ornithologists read: The American Naturalist, Animal Behaviour, The Auk, Behavioral Ecology, BMC Ecology (online only), The Condor, Ecology, Ibis, Journal of Avian Biology, Oecologia, Oikos, PLoS Biology (online only), and Proceedings of the National Academy of Sciences (biology articles only). To capture trends, we chose the years 2000 (but 2001 for BMC Ecology, the year it was first published), 2005, 2010, and 2015, recording pages of appendices and pages of SM for each article to evaluate whether SM were replacing appendices. Initially we reviewed every article in Animal Behaviour for 2015, but, because of time constraints, we thereafter subsampled only 2 randomly chosen issues within the specified years. Our initial forays revealed that many journals did not begin having SM until after 2000; thus, to avoid wasted effort, we began in 2005 for Behavioral Ecology, Oecologia, Oikos, Proceedings of the National Academy of Sciences, and PLoS Biology. We reviewed only full papers that reported original data; commentaries, reviews, and short communications were not considered. We evaluated trends with general linear models where the response variable was either the log of the number of pages of appendices + 1 or the log of the number of pages of SM + 1, year was a class explanatory variable, and log of the number of pages in an article was a covariate.

In total we reviewed 1,876 articles, of which 262 (14.0%) had appendices and 373 (20.0%) had SM. Although there were significant temporal trends in numbers of pages of appendices for 7 journals, there was no clear, overall pattern of decreases or increases (Table 1). By contrast, 10 journals had statistically significant positive trends in SM over time, including one exclusively online journal (Table 1). These findings are consistent with previous studies on different sets of journals (Schaffer and Jackson 2004, Evangelou et al. 2005, Schriger et al. 2011, Kenyon and Sprague 2014, Rafferty et al. 2015, Williams 2016); in fact, in some cases, journals now have SM associated with 100% of their articles (Borowski 2011, Pop and Salzberg 2015). SM were less often found in ornithological journals, particularly in North America: 1 of 156 (0.6%) articles in The Auk, 0 of 150 articles in The Condor, 21 of 116 (18.1%) articles in Ibis, and 9 (all in 2015) of 111 (8.1%) articles in the Journal of Avian Biology. Finally, for 4 journals, we found evidence that as pages of appendices decreased, pages of SM increased (Table 1).

TABLE 1.

Comparison of journal use of appendices and supplementary materials evaluated for a subset of articles from 2000, 2005, 2010, and 2015. Statistics are from general linear models where year was a class explanatory variable, the total number of pages we reviewed for a journal in a year was a covariate, and pages of appendices or pages of supplementary materials were the response variables. A dash (–) indicates that there were no instances of the response variable. Letters denote least square means (LS-means) comparisons of years in temporal sequence; if there are only 3 letters, 2000 was not evaluated. A's are associated with the highest means, followed by B, followed by C. Years sharing letters did not differ in numbers of pages devoted to appendices or supplementary materials.

Another aspect of SM is content; with the exceptions of audio and video files, allocation of types of materials within appendices and SM was broadly similar (Figure 1). Schriger et al. (2011) found that the number of tables was the fastest-growing content in SM; we did not evaluate trends.

FIGURE 1.

For a subset of articles, percentages of each type of content found in 262 articles (white bars) that had appendices and 373 articles (black bars) that had supplementary materials.

Costs

The first cost of SM is to authors. In the past, authors summarized relevant data as succinctly as possible in their papers, and filed data of less relevance. With the option of SM, authors now must judge whether less relevant data are worth sharing with readers (Kenyon and Sprague 2014, Pop and Salzberg 2015). If authors opt for SM, they must then invest in the organization and production of these materials (Carpenter 2010), investments that may be more than trivial. To reduce these investments, authors may not follow due diligence, which may result in poor-quality SM. Moreover, if authors invest insufficiently in assessing the value of SM, irrelevant and overly detailed information could become entrenched in a burgeoning digital archive; this would be detrimental to science (Pop and Salzberg 2015). As an illustration, in a recent review of publicly mandated data-archiving, Roche et al. (2015) reported that >50% of data may not be usable. Another cost, on the flip side, is that busy readers may deem SM of limited value or too costly, in terms of time, to access (Rosenthal and Reich 2010); thus, authors may invest significant effort for low return. Formatting inconsistencies among journals are an irritation for many of us who publish; reformatting each time one resubmits a paper is painstaking. Now, with SM, a whole new set of guidelines may have to be reviewed; these may include size limits, file formats, and so on (NISA/NSAIS 2013). All of this adds additional time to the submission process. Also, if a reviewer decides to pursue a link that doesn't work (see below), this can delay peer review (Brooks and Markwell 2008). At one extreme, ubiquitous inclusion of SM in some journals implies that SM are required, which places additional burdens on authors, reviewers, production teams, and readers.

The second cost of SM is to reviewers. Some journals request that SM be reviewed along with associated articles (Pop and Salzberg 2015). Reviewers may refuse, arguing against further burdening what is essentially volunteerism. If reviewers can be convinced to evaluate SM, it could increase turnaround times on papers (particularly if a link doesn't work because it was entered improperly), which is neither in authors' nor in journals' best interests (Brooks and Markwell 2008, Borowski 2011). Reduced motivation for authors to invest in editing SM, and reluctance by reviewers to vet, can lead to poor-quality SM of questionable scientific merit. Maunsell (2010) cited these issues in an editorial in which her journal announced the end of SM. By contrast, Marcus (2009) argued that any limit to SM is arbitrary. We suspect that there is a continuum in how researchers view this.

The third and perhaps most significant cost of SM is to production teams (editors and copyeditors). One of the biggest costs to journals can be for copyediting, and there is always pressure to reduce costs. Thus, production teams may decide not to review SM (Kenyon and Sprague 2014). If we layer this atop inadequate attention from authors and reviewers, it presents another threat to the quality of SM. An additional production cost is for curation of SM, but this may not be a dire expense, given that digital storage space continues to come down in price.

As we prepared this commentary, we became most acutely aware of the fourth cost of SM, which is to readers. First, navigating SM can be extremely time consuming (see Anonymous 2015). For example, there is significant variation in how one accesses SM at different journals; one may have to scroll to find links, and multiple links are sometimes required to get to SM. Second, if storage space is not maintained (e.g., because a journal goes bankrupt, leading to link rot) or if uniform resource locators (URLs) are entered incorrectly (see above), dead links may make it impossible to retrieve SM (Anderson et al. 2006, Aronsky et al. 2007, Brooks and Markwell 2008, Nagaraja et al. 2011). Third, the sheer volume of material can be staggering, making it difficult or impossible to extract what is relevant (Pop and Salzberg 2015). It is already a daunting challenge to stay atop even a subset of the literature; this is a truly dark side of cheaply accessible information. In the journals we searched, it was not unusual to find dozens of pages of SM per paper (Figure 2). In an added irony, we found reference to SM within SM! Pop and Salzberg (2015) cite a few particularly amazing extremes: 165 pages of SM for a 6-page paper, and 144 pages of SM for a 5-page paper. The SM in the latter case had multiple sections, with authors listed in different orders. Fourth, SM can be a minefield of storage formats, and in some instances require downloading unfamiliar software with which a reader has to become conversant (Aronsky et al. 2007, Pop and Salzberg 2015). Fifth, one never knows whether SM are worth tracking down, for reasons already given. Sixth, readers who prefer to work from hard copies of papers or from print subscriptions lose that option when they have to return to the Internet to acquire SM. SM becomes unavailable to readers who venture to places where Internet access is unavailable or too expensive. In these circumstances, even if one works from on-screen versions of papers, access to potentially relevant SM is lost. Seventh, it is not uncommon to find that authors refer within the paper to only a fraction of associated SM links (Pop and Salzberg 2015). Do we allow our students to leave tables and figures uncited in their lab reports?

FIGURE 2.

Frequency distribution (excluding 1,433 zeros) of the numbers of pages of supplementary materials for 443 articles we evaluated.

A somewhat distinctive cost of SM is that they may contain citations that do not appear in an associated paper; these citations may not be captured by search engines—and, hence, authors may not be credited for their contributions to research (Rafferty et al. 2015). This may affect a researcher's competitiveness in seeking positions or promotions, or in applying for grants. However, this becomes a problem only if there are biases in the kinds of research that are relegated to SM. Nonetheless, this latter trend should continue to be monitored.

Benefits

We live in an era of phenomenal growth in data acquisition. In the past, this would have created a dilemma for authors because large volumes of data can be difficult to summarize. Now, authors have the option of providing many more summaries of information than may be necessary in a paper, but that may be of interest to part of a paper's readership; authors have virtually unlimited capacity to link information they deem relevant. This may include raw data and programs to evaluate the data in the same manner as the authors. Retaining all this information in a paper may impede interpretation for the majority of its audience. Second, journals' adoption of SM has freed authors (in some cases) from storing those materials on their personal websites; this is likely to reduce link rot (Anderson et al. 2006). Third, a variant of the file-drawer problem— wherein only statistically significant results are presented in detail—may be addressed by SM. Authors now have the option of depositing statistically nonsignificant results in SM. Fourth, data can be hard-won, and even if the data have no immediate purpose, it would be wasteful to discard them, assuming they are of reasonable quality (although such data may be deposited in locations other than SM; e.g., Dryad,  http://datadryad.org/). Long-term repositories may pay substantial dividends; indeed, many who have had to provide their data to accompany publications have received notice that someone is using the data to support additional research.

A potential benefit to reviewers of SM is that they can access more information than is provided in a terse paper, possibly facilitating their ability to evaluate scientific merit (Borowski 2011). On the other hand, if freed from reviewing SM, a reviewer may need less time to complete a review of a paper that would otherwise have contained volumes of peripheral information.

Production teams may incur substantial benefits from SM if they are not tasked with formatting the materials; this is often the case with some raw data that appear in SM (e.g., csv or mpeg files; Carpenter 2009). Indeed, SM appear in a variety of forms that have limited or inconsistent formatting. An additional benefit, primarily for print journals, is that costs of printing are saved, and these can be substantial for color images or voluminous SM (Evangelou et al. 2005, Carpenter 2009). Finally, the more SM a journal produces, the greater its web presence. However, we are not convinced that this latter perk will have important consequences in the mushrooming cloud of web volume.

One of the biggest benefits to readers may be from having access to SM that are too expensive to produce in print, such as high-quality images, or SM that cannot be produced in conventional print format, such as sound and video files (Figure 1). Of all the SM we encountered, sound and video files had the most immediate impact for the least investment on our part. A second benefit to readers is flexibility in how one reads a paper. Granted, one can always skip sections, but if all SM appeared within the body of the paper, sheer length might discourage readers from even attempting to read it, and readers might become distracted from the main thread. A paper that has only the barest essentials may flow much better (Pop and Salzberg 2015).

Challenges and Prognoses

We believe that there is a place for SM, but it needs to be carefully considered at every step, and this creates dilemmas for everyone involved. If we don't confront these dilemmas, we risk accepting poor-quality SM that will deter scientific advances. As we skimmed SM in >400 papers, we observed the gamut of inaccessible to wonderful SM. An optimistic view is that wonderful SM will become the norm and will follow guidelines provided by NISO/NFAIS (2013), including recommendations for improving links. All SM links should appear in both the print and online versions of a paper; be cited within the paper where the information is relevant, as well as at the end of the paper; and be included in the table of contents (Williams 2016). For readers who disconnect, it might be valuable if the default were for SM to download along with the paper (Marcus 2009, Laue 2010), which would alert readers who want to print only the paper to follow attendant instructions. One reviewer of this commentary suggested that journals indicate whether their SM have been peer reviewed, a view we share.

One can only wonder about the future of scientific publishing and whether this is all moot. Will there always be publications, or will we be carrying around multimedia, multidimensional synopses of research with interactive data interfaces (Carpenter 2009)? Brave new world.