Many commentators have conjectured about the nature and promise of future scholarly electronic journals, and some have even predicted the eventual demise of the journal in its present paper form. [1] However, as Ann Okerson wrote, "One can fantasize endlessly about electronic 'journals,' but without active authorship and readership there is nothing." [2] Even beyond this, e-journals must be used. "Use" implies not only reading e-journals and contributing to them, but building on the findings reported--it implies that the research reported in e-journal articles have an impact on scholars and researchers in the discipline which the e-journal represents. If e-journals have little or no impact on research and researchers, they cannot play an important role in scholarly communication.
This article reports hard empirical data on the impact of the first wave of e-journals on the scholarly communities they serve. It assesses the extent to which scholars and researchers are aware of, are influenced by, and build their own work upon research published in e-journals. It does this by examining the artifacts of scholarly communication--the journal article and the references it makes.
A citation analysis was conducted for 39 scholarly journals that began electronic publication no later than 1993. Citation data for these journals were tabulated and analyzed. For journals that publish both print and electronic versions, citations to articles published prior to parallel publication were eliminated. The eight most highly cited e-journals were identified. Citation and publication data for three high ranking e-journals in the study were compared to similar data for print journals in the same fields. The seven most highly cited articles from the e-journals in the study were determined.
This study is part of a larger project [3] that studied several of the characteristics of e-journals, including demographics (e.g., number of articles published, publication frequency, language, discipline, and charging policy), access problems and issues, references, and the citation analysis findings reported here. Selected findings from the larger study, [4] and an examination of access problems and issues [5] are discussed in separate papers.
Citation analysis is a generic term for a set of well-known techniques that have a long history in bibliometric studies of scholarly communication. [6] As artifacts of the scholarly communication process, citations can reveal formal communication patterns. Methods of citation analysis are unobtrusive and can be highly reliable.
Citation-based measures are not the only, or even necessarily the best, measures of such impact. There are other approaches to evaluating the impact of e-journals. Publishers can gain some information regarding use by noting the number of subscriptions to their journals or by counting the number of times articles are accessed or downloaded from host servers. These data provide useful indicators of one type of use, but they fail to assess the extent to which e-journal articles play a role in scholarship, research, and the advancement of knowledge.
Citation analysis assumes that references to a particular journal article reflect a scholarly impact of that article on the author of the citing work. It further assumes that the accumulated total of citations to a given author's works in some sense reflects the impact of that author on scholarship and research.
The same assumption applies to the assessment of the impact of journals--that the accumulated total of citations to all articles published by a journal is an indicator of the impact of that journal on the relevant discipline(s). Many studies have used citation-based measures as evaluation instruments for journals, research performance, university departments, universities, and published works. [7]
The main defect of citation analysis lies in the extent to which the primary assumption underlying all approaches to citation indexing--that citation reflects scholarly impact--is false. While the meaning of citation has been debated, [8] citation analysis is generally regarded as a valuable tool for determining the impact of scholarly works.
Citation analyses are normally conducted on the references of thousands of print (and a few electronic) journals. The Institute of Scientific Information (ISI) publishes three citation indexes--Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI)--that allow searches of journal names as cited works.
When a cited work search on an e-journal such as The Public-Access Computer Systems Review (PACS Review) is conducted on SSCI, one retrieves bibliographic citations to all the articles in journals indexed by SSCI that have cited a PACS Review article in one or more reference.
ISI represents the thousands of scholarly journals it indexes as the most significant journals in the various fields and disciplines of scholarship. Thus, a cited reference search should retrieve the majority of all citations made in scholarly journals to the cited work by the most important scholarly papers. It will not reveal citations to a work made in less significant journals or in books, dissertations, or other materials. In the context of the World-Wide Web, a cited work search will not reveal the hypertext links (analogous to citations) to works that are made in Web pages. However, these limitations notwithstanding, the ISI indexes are widely recognized as providing a valuable measure of the formal scholarly impact of articles, journals, and authors.
The following steps were taken to select a set of e-journals for study and to generate citation statistics for them:
All known scholarly, peer-reviewed e-journals with a reasonable chance of being cited were selected for study.
The time lag in the publication schedules of print journals is substantial, and many months are needed to provide time for a journal article to be read, to become part of an author's thought, and to be cited in a research paper. For these reasons, the study was limited to those scholarly, peer-reviewed e-journals that began electronic publication no later than 1993 (many well prior to 1993). There were thirty-nine such e-journals. Appendix A lists the e-journals in the sample and the year in which they began electronic publication. The journals were drawn from two directories of e-journals; detailed procedures for selecting the sample are reported elsewhere. [9]
JCR is a tool published by the Institute for Scientific Information that includes several citation-based measures of journal impact for the source journals indexed. These measures will be discussed later. The first step in the present research was to check in JCR to see if the e-journals in the sample were included; only the Bulletin of the American Mathematical Society was there. To perform citation-based analyses on the remaining 38 e-journals, it was necessary to conduct cited work searches on the three ISI databases: SciSearch, Social SciSearch, and Arts & Humanities Search.
All cited work searches were done on the DIALOG system. DIALOG's OneSearch was used to search all three of the ISI citation databases simultaneously. This made it unnecessary to try to predict which database(s) should be searched for each journal (a problem for e-journals with strong interdisciplinary connections). (It is conceivable, for example, that the e-journal Psycoloquy might be cited by an article in a science journal or even by a humanities journal.)
The cited work field, as found in the ISI databases loaded on DIALOG, is a phrase-indexed field of at most 23 characters, the first three of which are "CW=." Thus, there are only 20 characters in which to represent the title of a given journal (or book). Finding all of the forms in which a title like Psyche: An Interdisciplinary Journal of Research on Consciousness appears in the databases is the most challenging part of doing a cited work search. For Psycoloquy, seven different forms were found (see Table 1). Note the several abbreviations and misspellings.
CW=PSYCOLOQUY CW=PSYCOLOQUY ELECTRONI CW=PSYCOLOQUY REFEREED CW=PSYCHOLOQUY CW=PSYCHOLOQUY DEC CW=PSYCHOLOQUY REFEREED CW=PSYCHOLOQUY REFERRED
To conduct a cited work search on DIALOG, one must first EXPAND on reasonable character strings, and then SELECT apparent hits. The EXPAND command opens a window to a section of the alphabetically arranged index entries, from which likely entries can be spotted and selected.
For example, consider the Journal of Statistics Education, a relatively simple title for which to search. The following considerations are relevant to one's strategy.
The procedures followed in the ISI databases for representing journal names are inconsistent. In particular:
The result of using the EXPAND command on the character string "CW=J STAT" is given in Table 2. Twelve index entries alphabetically related to "J STAT" are displayed in response to the command.
? E CW=J STAT Ref Items Index-term E1 1 CW=J STARUCT ENG ASCE E2 1 CW=J STAST COMPUT SIMUL E3 2 *CW=J STAT E4 1 CW=J STAT ANAL E5 13 CW=J STAT ASS E6 1 CW=J STAT ASS SEP E7 5 CW=J STAT COMP E8 1 CW=J STAT COMP S E9 6 CW=J STAT COMP SIM E10 1 CW=J STAT COMP SIMU E11 5 CW=J STAT COMP SIMUL E12 1 CW=J STAT COMP SIMULA Enter P or PAGE for more ? P Ref Items Index-term E13 2 CW=J STAT COMP SIMULAT E14 2 CW=J STAT COMP SIMULATI E15 1 CW=J STAT COMPUT E16 1 CW=J STAT COMPUT SCI E17 542 CW=J STAT COMPUT SIM E18 16 CW=J STAT COMPUT SIMUL E19 3 CW=J STAT COMPUT SIMULA E20 1 CW=J STAT COMPUT STIMUL E21 6 CW=J STAT COMPUTATION S E22 5 CW=J STAT COMPUTER SIMU E23 1 CW=J STAT COMPUTG SIMUL E24 1 CW=J STAT HPYS
EXPAND commands were given for several forms of this e-journal name, and the results were carefully reviewed for instances that might represent the Journal of Statistics Education (see Table 3).
CW=J STATISTICS CW=J OF ST CW=JSE CW=JOURNAL ST CW=JOURNAL OF ST CW=J STAT CW=JOURN STAT CW=JOURN OF STAT CW=JOUR OF ST CW=JOUR STAT
This example demonstrates the problematic nature of conducting cited work searches on the ISI databases. A "good" search depends on trying all reasonable combinations of words and abbreviations. Errors are possible, even inevitable, despite the best efforts of the searcher. Thus, there is no guarantee that the results of the cited work searches to be presented here are complete. However, as the example demonstrates, a great deal of effort was expended in trying many combinations for each e-journal.
When all forms of a given e-journal were located, they were combined into a single set. The RD (remove duplicates) command was then used, because the searches were conducted in three ISI databases simultaneously and some citing journals are indexed by more than one ISI database. Using DIALOG's KWIC feature, bibliographic citations for the citing articles were printed, as well as the portion of the bibliography that included the cited reference(s). This was often necessary to verify that the cited journal was in fact the journal that was sought and to identify the cited author(s) and year(s).
A "citation" is interpreted by ISI as a unique reference to a work in a citing article. A given citing article (called a "source article" by ISI) may include several different references to articles in the same journal. Each of these is counted as a separate citation. Thus, the number of articles citing a work must be less than or equal to the number of citations to the work. Typing the citing articles in the KWIC format reveals all the citations. The number of citations can also be counted directly, without using KWIC, by using the DIALOG command SET POSTINGS ON and displaying the relevant retrieval set. [10]
Besides the form-of-name difficulty, there were other problems with some of the e-journals. Some e-journals with print counterparts were publishing in print form before they became electronic, and they are currently publishing two parallel versions. In order to eliminate citations to articles published only in print form, this study excluded citation data from these journals prior to the start of parallel print and electronic publication.
Another problem with a few e-journals was the difficulty in separating out the e-journal citations from citations to other books or journals of the same name. The worst example of this problem was Psyche. Psyche is the name of at least two other journals and many books. A cited work search on "CW=PSYCHE" retrieved more than 5,000 citing articles. If the publication dates were limited to the years in which the e-journal Psyche was published, there were still more than 600 citing articles. Without a great deal of additional work and significantly greater cost, including finding and inspecting the original citing articles in many cases, it was impossible to limit retrieval to the e-journal Psyche. This was not done in this instance, and the number of times the e-journal Psyche has been cited is unknown. The findings to be presented here do not include Psyche. The same problem was also experienced with a few other e-journals, but on a much smaller scale, and these ambiguities were resolved.
The citation data reported here were gathered during February 1996. Eleven of the 39 e-journals for which cited work searches were conducted have print counterparts, and are thus published in "mixed," or parallel, formats, while the remaining 28 e-journals are "pure"--they are only published in electronic form. It is not possible from ISI's citation data (nor, indeed, from most bibliographic citations) to determine whether it is the print version or the electronic version of an e-journal article that was read and cited by an author.
Since the purpose of this study is to measure the impact of electronic journals, the data for these eleven e-journals must be treated with great caution. The chances are very good that the great majority of citations to most of these e-journals were generated by the print version of the journal rather than the electronic version. Although every scholar reads print journals, not every scholar reads electronic journals. Thus, for e-journals with a print counterpart, the chances are much higher that a given article was discovered, read, and cited as a result of its appearance in print than from its publication in electronic form.
The total number of citations tends to measure the cumulative effect of a journal, since its articles can generate citations and credit for the journal for decades after publication. It is a long-term measure, clearly biased in favor of older journals that have been publishing for many years. Another characteristic of this measure is that, all other things being equal, it favors journals that publish many articles per year over those that publish only a few. The number of articles published in a given year will clearly affect the total number of citations received in subsequent years.
Table 4 provides a frequency distribution of the citation data--the total number of citations to each e-journal in the sample.
Total Number of
Citations to E-Journal** |
Number of E-Journals |
---|---|
0 | 15 |
1-5 | 13 |
6-10 | 3 |
11 | 1 |
... | ... |
14 | 1 |
... | ... |
35 | 1 |
... | ... |
38 | 1 |
... | ... |
111 | 1 |
... | ... |
190 | 1 |
... | ... |
> 1,500 | 1 |
*The number of citations to Psyche could not be determined.
**For e-journals with print counterparts, the number of citations includes all citations to articles published in the journal during the period the journal was published electronically.
A few e-journals are notable exceptions. Eight have been cited ten or more times. These e-journals are listed in Table 5, along with their disciplines or fields, whether or not the e-journal has a print counterpart, and the total number of times the journal has been cited. Three of the eight e-journals, and three of the top five, are also published in print form.
Among the top eight e-journals, three are from the sciences, five are from the social sciences, and none are from the humanities.
Name of E-Journal | Discipline | Print Version? | Number of Citations To the E-Journal* |
---|---|---|---|
Bulletin of the American Mathematical Society [11] | Mathematics | Yes | > 1,500 |
Online Journal of Current Clinical Trials [12] | Medicine | No | 190 |
The Public-Access Computer Systems Review [13] | Library and information science | (Yes) | 111 |
Digital Technical Journal [14] | Computer science | (No) | 38 |
Psycoloquy [15] | Psychology | No | 35 |
Interpersonal Computing and Technology [16] | Effects of technology on society | No | 14 |
Electronic Journal of Communication [17] | Communication | No | 11 |
Postmodern Culture [18] | Modern culture | No | 10 |
* For the Bulletin of the American Mathematical Society, data are for 1994 only. Data for all other journals include the total of all citations to the journal since publication began in electronic form to the present.
The number of citations to journals in a discipline varies widely from field to field, a point made by Eugene Garfield [21] and since demonstrated in other studies. [22] Thus, it is unfair and misleading to compare the citation productivity of authors and journals in different fields (e.g., to draw inferences about quality or impact).
To put the numbers in Table 5 in perspective, we must make comparisons to print journals in the same fields. Another Institute for Scientific Information product, Journal Citation Reports (JCR), is available for such a comparison. JCR compiles and reports several citation statistics for the top journals in all fields.
As noted earlier, among the 39 e-journals in the sample, only the Bulletin of the American Mathematical Society appeared in the Journal Citation Reports lists, which necessitated the DIALOG searches on the three ISI databases reported earlier. Some of the e-journals in the sample--especially the eight listed in Table 5--are seemingly candidates for addition to Journal Citation Reports.
Table 6 reports JCR data for six fields in which some of the most highly cited e-journals are published: communication, computer science, information and library science, mathematics, medicine, and psychology. The JCR on CD-ROM (Version 1.0 for Windows; published in 1995) was used to produce these data. The JCR is in two parts, one part for the social sciences (1,402 journals) and the other for the sciences (4,513 journals). The "filter" feature was used to create a category of journals in each of the six fields. The journals in each category were then ranked by total number of citations received in 1994 (the last year for which JCR citation data were available) and printed. It was then easy to determine the top-and bottom-ranked journals, the three quartiles, and other percentiles.
As would be expected from the fact that the numbers of journals and researchers in each field vary considerably, Table 6 shows that the number of citations received by the median journal in each discipline differs widely from field to field. Citation peculiarities and practices of the fields are probably different as well. The median journal in library and information science, for example, received only 54 citations in 1994. The factors by which library and information science journals must be multiplied to make the median journals equivalent in terms of total number of citations received are: communication (3.6), computer science (3.2), mathematics (7.0), medicine (8.5), and psychology (4.4). The other quartiles also differ significantly from field to field.
Name of Field and Number of Journals in Ranking (N) | First Quartile Journal | Median Journal | Third Quartile Journal |
---|---|---|---|
Communication (N=23) | 381 | 194 | 132 |
Computer science (N=205) | 538 | 172 | 65 |
Library and information science (N=59) | 136 | 54 | 30 |
Mathematics (N=202) | 751 | 379 | 197 |
Medicine (N=114) | 1,632 | 461 | 202 |
Psychology (N=99) | 717 | 240 | 96 |
Consequently, Table 6 is offered only to give the reader a general sense of how often journals in various fields are cited and to provide a very rough yardstick against which the impact of the e-journals in the sample can be measured. Having said this, if one compares the citation counts in Table 5 with those in Table 6, it is clear that most e-journals are having very little impact on formal scientific and scholarly communication.
For these reasons, ISI created another citation-based measure of effectiveness, the impact factor, which normalizes the citations received by journals and looks only at the previous two years of publication.
The potential impact on scholarship and research is clearly a function of the number of articles published. If a journal publishes only a few articles each year, it cannot influence many other researchers and writers, and it will not be cited very often. The long-term impact of a journal can be assessed by the total number of citations received overall, or in a given year. The impact factor, somewhat misnamed, is defined by ISI as an average. Table 7 shows the formula for computing the 1995 impact factor for a given journal.
Impact factor for 1995 = (Number of citations received in 1995 to 1993 and 1994 articles) ------------------------------------------------------ (Number of articles published in 1993 and 1994)
The immediacy index measures the extent to which articles make a quick impact on readers--the timeliness or currency of the journal. Historical journals would presumably have low immediacy indexes; cutting edge medical journals would have relatively high immediacy indexes. One would expect e-journals to have high immediacy indexes, since speed of publication is one of the most often cited advantages of e-journals. Table 8 shows the formula for computing the 1995 immediacy index for a given journal.
Immediacy index for 1995 = (Number of citations received in 1995 to 1995 articles) ------------------------------------------------------- (Number of 1995 articles)
Among the eight most highly cited e-journals identified in Table 5, the Bulletin of the American Mathematical Society and the Digital Technical Journal both had parallel print and electronic versions for most or all of their electronic history, resulting in the previously discussed problem of sorting out which citations are to the electronic versions. The last three e-journals listed in Table 5--Interpersonal Computing and Technology, the Electronic Journal of Communication, and Postmodern Culture--had far fewer citations than the others. These five e-journals were not studied any further.
Three e-journals were chosen for more detailed study: the Online Journal of Current Clinical Trials, Psycoloquy, and The Public-Access Computer Systems Review. Using the effectiveness measures presented in Table 9, 1994 JCR data for medicine, psychology, and library and information science journals were compared to 1995 data for OJCCT, Psycoloquy, and the PACS Review, respectively.
Measure | Journal Characteristic Measured |
---|---|
Total number of citations received in a given year | Long-term, aggregate impact of a journal in the year that the measurement is taken. |
Impact factor | Roughly, the average number of times an article is cited in the two years following publication. |
Immediacy index | Extent to which articles are cited quickly. |
Number of articles published in a given year | Potential impact of a journal in that year. |
Table 10 compares 1994 JCR citation data for 114 medical journals (JCR category "Medicine, General & Internal") with 1995 citation data for the Online Journal of Current Clinical Trials. The measures reported are total citations, impact factor, immediacy index, and total number of articles published. For each measure, the data for the first-ranked journal, the median journal, the last-ranked journal, and data from OJCCT are reported. All but the OJCCT data are taken from the 1995 edition of Journal Citation Reports on CD-ROM.
The final column in Table 10 reports the percentile rank of OJCCT when compared to the 114 medical journals indexed by Journal Citation Reports. In total citations, for example, the top ranked journal (The New England Journal of Medicine) was cited nearly 100,000 times in 1994. With 82 citations, OJCCT is well below the median medical journal, and it ranks at the 9th percentile.
On the other hand, the percentile rankings for both impact factor and immediacy index for OJCCT are comparatively high, at 88 and 75 percent respectively. Thus, the average article in OJCCT is cited more than 88 percent of the e-journals indexed by JCR. The reason for the relatively small total number of citations received is that OJCCT has not published very many articles--only four in 1995.
Measure | First-Ranked Journal (1994)* | Median Journal (1994) | Last-Ranked Journal (1994) | OJCCT (1995) | OJCCT % Rank** |
---|---|---|---|---|---|
Total citations | 98,940 | 461 | 4 | 82 | 9 |
Impact factor | 22.7 | .368 | .01 | 2.54 | 88 |
Immediacy index | 11 | .247 | 0 | .25 | 75 |
Total articles | 785 | 107 | 0 | 4 | 7 |
* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, The New England Journal of Medicine, The New England Journal of Medicine, WHO Technical Report Series, and the British Medical Journal.
** Percentiles were computed by comparing 1995 OJCCT data to JCR data for 114 journals in medicine in 1994, the last year these data were available.
The results for Psycoloquy shown in Table 11 are quite similar to those of OJCCT--low percentile rankings for total citations and total articles, but a relatively high impact factor. While the average article published in Psycoloquy was cited more than twice as often as the median psychology journal, it published far fewer articles than the typical journal. Thus, its overall impact, as measured by total number of citations, is relatively small. The immediacy index for Psycoloquy was 0.0--no 1995 article was cited by an ISI-indexed journal in 1995. However, Psycoloquy's scholarly skywriting represents a rapid scholarly reaction that is different from, but in some sense equivalent to, citation.
Measure | First-Ranked Journal (1994)* | Median Journal (1994) | Last-Ranked Journal (1994) | Psycoloquy (1995) | Psycoloquy % Rank** |
---|---|---|---|---|---|
Total citations | 9,632 | 240 | 2 | 27 | 7 |
Impact factor | 7.19 | .415 | .029 | .94 | 76 |
Immediacy index | 1.5 | .083 | 0 | 0 | 27*** |
Total articles | 510 | 30 | 0 | 4 | 7 |
* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, Psychological Bulletin, Psychological Review, Behavioral and Brain Sciences, and Psychological Reports.
** Percentiles were computed by comparing 1995 Psycoloquy data to JCR data for 99 journals in psychology in 1994, the last year these data were available.
*** Tied for last with 23 other journals.
Table 12 examines The Public-Access Computer Systems Review. Compared to other journals in its field, the PACS Review enjoys the highest percentile rankings of all the e-journals studied. It was cited nearly as often in 1995 as the median journal in library and information science. It also had a high impact factor and a very high immediacy index score. These rankings were achieved in spite of the fact that the total number of articles published in 1995 was only at the 15th percentile. On the basis of these comparisons, it can be argued that the PACS Review is the most successful of all the e-journals examined in this study.
Measure | First-Ranked Journal (1994)* | Median Journal (1994) | Last-Ranked Journal (1994) | PACS Review (1995) | PACS Review % Rank** |
---|---|---|---|---|---|
Total citations | 725 | 54 | 1 | 39 | 37 |
Impact factor | 1.314 | .263 | 0 | .43 | 71 |
Immediacy index | 1.143 | .021 | 0 | .2 | 86 |
Total articles | 109 | 22 | 0 | 5 | 15 |
* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, the Journal of the American Society for Information Science, College and Research Libraries, the Journal of the American Medical Informatics Association, and Library Journal.
** Percentiles were computed by comparing 1995 PACS Review data to JCR data for 59 journals in library and information science in 1994, the last year these data were available.
Using the citation data already presented, it was relatively easy to identify the most highly cited e-journal articles. All e-journals except BAMS were included in this analysis. The most highly cited e-journal articles are listed in Table 13. All are from just two journals, OJCCT and the PACS Review.
The most highly cited e-journal article is by J. M. Elwood, "The Effectiveness of Breast Cancer Screening by Mammography in Younger Women," which was published in OJCCT. It had earned 59 citations as of February 1, 1996. It is worth noting that the Elwood article has been cited more often than all but three of the e-journals studied. In fact, each of the seven articles has been cited more often than 33 of the 39 e-journals studied. The seven articles account for 151 of the 444 citations received by all the e-journals together (again excluding BAMS).
No. of Citations (2/1/96) | Article |
---|---|
59 | Elwood, J. M., B. Cox, and A. K. Richardson. "The Effectiveness of Breast Cancer Screening by Mammography in Younger Women." Online Journal of Current Clinical Trials, 25 February 1993, doc. no. 32. |
26 | Villari, P., C. Spino, T. C. Chalmers, J. Lau, and H. S. Sacks. "Cesarean Section to Reduce Perinatal Transmission of Human Immunodeficiency Virus. A Metaanalysis." Online Journal of Current Clinical Trials, 8 July 1993, doc. no. 74. |
14 | Dickersin, K., and Y. I. Min. "NIH Clinical Trials and Publication Bias." Online Journal of Current Clinical Trials, 28 April 1993, doc. no. 50. |
14 | Harnad, Stevan. "Post-Gutenberg Galaxy: The Fourth Revolution in the Means of Production of Knowledge." The Public-Access Computer Systems Review 2, no. 1 (1991): 39-53. [23] |
14 | Klein, M. C., R. J. Gauthier, S. H. Jorgensen, J. M. Robbins, J. Kaczorowski, B. Johnson, M. Corriveau, R. Westreich, K. Waghorn, M. M. Gelfand et al. "Does Episiotomy Prevent Perineal Trauma and Pelvic Floor Relaxation?" Online Journal of Current Clinical Trials, 1 July 1992, doc. no. 10. |
12 | Okerson, Ann. "The Electronic Journal: What, Whence, and When?" The Public-Access Computer Systems Review 2, no. 1 (1991): 5-24. [24] |
12 | Wiggins, Rich. "The University of Minnesota's Internet Gopher System: A Tool for Accessing Network-Based Electronic Information." The Public-Access Computer Systems Review 4, no. 2 (1993): 4-60. [25] |
The citation data reported in this study show that the great majority of scholarly, peer-reviewed e-journals have had essentially no impact on scholarly communication in their respective fields. Only eight of the 39 e-journals studied have been cited ten or more times over their lifetimes. Given that eleven of the e-journals have print counterparts, these findings are especially telling. The top eight e-journals are from the sciences, medicine, and social sciences.
To be meaningful, comparisons between journals must be for journals in the same field, since disciplines vary widely among themselves in their citation practices. Using Journal Citation Reports data, the top three e-journals--OJCCT, the PACS Review, and Psycoloquy--were compared to other journals in medicine, library and information science, and psychology, respectively. All three e-journals ranked highly in impact factor, which measures the impact of the typical article during the two year period following publication. OJCCT and the PACS Review also ranked well above the median journals in their fields on the immediacy index, which is a measure of the speed with which published articles are cited. The PACS Review also earned a respectable 37th percentile ranking in total citations. All three e-journals ranked poorly in total number of articles published.
Based on these findings, I conclude that the Online Journal of Current Clinical Trials, The Public-Access Computer Systems Review, and Psycoloquy all show great promise of becoming one of the top journals in their respective fields. However, all are publishing far fewer articles in a given year than most other journals in their disciplines. Thus, while their high impact factors for a typical article show great promise, the overall scholarly impact of these e-journals on their disciplines is not great. Indeed, they cannot have a major impact until they publish many more articles annually than they presently do, while maintaining the present overall high quality of their articles. More authors will need to view e-journals as legitimate publication vehicles before e-journals can assume a significant role in the scholarly communication process.
1. For example, see: Andrew M. Odlyzko, "Tragic Loss or Good Riddance? The Impending Demise of Traditional Scholarly Journals," International Journal of Human-Computer Studies 42 (1995): 71-122.
2. Quoted in: M. P. Collins and Z. L. Berge, "IPCT Journal: A Case Study of an Electronic Journal on the Internet," Journal of the American Society for Information Science 45 (1994): 771-776.
3. I am grateful to the Online Computer Library Center, Inc. (OCLC) for partial funding of the overall study.
4. Stephen P. Harter and Hak Joon Kim, "Electronic Journals and Scholarly Communication: A Citation and Reference Study," in Proceedings of the ASIS Midyear Meeting (San Diego: American Society for Information Science, 1996), 299-315. Also available at <URL:http://ezinfo.ucs.indiana.edu/~harter/harter-asis96midyear.html>.
5. Stephen P. Harter and Hak Joon Kim, "Accessing Electronic Journals and Other Epublications: An Empirical Study," College & Research Libraries 57 (September, 1996), in press.
6. As excellent collection of such papers that includes an outstanding bibliography of additional sources is given in: Christine L. Borgman, ed., Scholarly Communication and Bibliometrics (Newbury Park, CA: Sage Publications, 1990).
7. For an early treatment of these applications, see: Francis Narin, Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity (Cherry Hill, NJ: Computer Horizons, Inc., 1976).
8. For example, see: T. A. Brooks, "Evidence of Complex Citer Motivations," Journal of the American Society for Information Science 37 (1986): 34-36; S. E. Cozzens, "What Do Citations Count? The Rhetoric-First Model," Scientometrics 15 (1989): 437-447; G. N. Gilbert, "Referencing as Persuasion," Social Studies of Science 7 (1977): 113-122; M. H. MacRoberts and B. R. MacRoberts, "The Negational Reference: Or the Art of Dissembling," Social Studies of Science 14 (1984): 91-94; and H. G. Small, "Cited Documents as Concept Symbols," Social Studies of Science 8 (1978): 327-340.
9. Harter and Kim, "Electronic Journals and Scholarly Communication."
10. I thank Peter Ingwersen for bringing my attention to the SET POSTINGS ON command for counting citations. This and other methods of conducting online citation analyses are described in: Finn Hjortgaard Christensen and Peter Ingwersen, "Online Citation Analysis: A Methodological Approach," Scientometrics 37 (September, 1996), in press.
11. See <URL:http://www.ams.org/journals/bull/>.
12. See <URL: http://www.oclc.org/oclc/promo/ejo_list.htm#cct>.
13. See <URL:http://info.lib.uh.edu/pacsrev.html>.
14. See <URL:http://www.digital.com/info/DTJ/home.html>.
15. See <URL:http://cogsci.ecs.soton.ac.uk/~harnad/psyc.html>.
16. See <URL:http://www.helsinki.fi/science/optek/>.
17. See <URL:http://cios.llc.rpi.edu/www/ejcmain.htm>.
18. See <URL:http://jefferson.village.virginia.edu/pmc/contents.all.html>.
19. The 1990 volume was published in book form in 1992; the 1991 volume in 1992; the 1992 volume in 1993; the 1993 volume in 1994; and the 1994 volume in 1996.
20. Stevan Harnad, "Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry," Psychological Science 1 (November 1990): 342-344. Also available at: <URL:ftp://cogsci.ecs.soton.ac.uk/pub/harnad/Harnad/harnad90.skywriting>.
21. Eugene E. Garfield, Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (New York: Wiley, 1979).
22. P. Vinkler, "Possible Causes of Differences in Information Impact of Journals from Different Subfields," Scientometrics 20 (1991): 145-161.
23. See <URL:http://info.lib.uh.edu/pr/v2/n1/harnad.2n1>.
24. See <URL:http://info.lib.uh.edu/pr/v2/n1/okerson.2n1>.
25. See <URL:http://info.lib.uh.edu/pr/v4/n2/wiggins1.4n2> and <URL:http://info.lib.uh.edu/pr/v4/n2/wiggins2.4n2>.
The year that the journal began electronic publication is shown in parentheses after the journal's name.
Stephen P. Harter, Professor, School of Library and Information Science, Indiana University, Bloomington, IN 47405. Internet: harter@indiana.edu.
The World-Wide Web home page for The Public-Access Computer Systems Review provides detailed information about the journal and access to all article files: <URL:http://info.lib.uh.edu/pacsrev.html>.
This article is Copyright © 1996 by Stephen P. Harter. All Rights Reserved.
The Public-Access Computer Systems Review is Copyright © 1996 by the University Libraries, University of Houston. All Rights Reserved.
Copying is permitted for noncommercial, educational use by academic computer centers, individual scholars, and libraries. This message must appear on all copied material. All commercial use requires permission.