Harter, Stephen P. "The Impact of Electronic Journals on Scholarly Communication: A Citation Analysis." The Public-Access Computer Systems Review 7, no. 5 (1996). (Refereed Article)

1.0 Introduction

Many commentators have conjectured about the nature and promise of future scholarly electronic journals, and some have even predicted the eventual demise of the journal in its present paper form. [1] However, as Ann Okerson wrote, "One can fantasize endlessly about electronic 'journals,' but without active authorship and readership there is nothing." [2] Even beyond this, e-journals must be used. "Use" implies not only reading e-journals and contributing to them, but building on the findings reported--it implies that the research reported in e-journal articles have an impact on scholars and researchers in the discipline which the e-journal represents. If e-journals have little or no impact on research and researchers, they cannot play an important role in scholarly communication.

This article reports hard empirical data on the impact of the first wave of e-journals on the scholarly communities they serve. It assesses the extent to which scholars and researchers are aware of, are influenced by, and build their own work upon research published in e-journals. It does this by examining the artifacts of scholarly communication--the journal article and the references it makes.

A citation analysis was conducted for 39 scholarly journals that began electronic publication no later than 1993. Citation data for these journals were tabulated and analyzed. For journals that publish both print and electronic versions, citations to articles published prior to parallel publication were eliminated. The eight most highly cited e-journals were identified. Citation and publication data for three high ranking e-journals in the study were compared to similar data for print journals in the same fields. The seven most highly cited articles from the e-journals in the study were determined.

This study is part of a larger project [3] that studied several of the characteristics of e-journals, including demographics (e.g., number of articles published, publication frequency, language, discipline, and charging policy), access problems and issues, references, and the citation analysis findings reported here. Selected findings from the larger study, [4] and an examination of access problems and issues [5] are discussed in separate papers.

2.0 Citation Analysis

Citation analysis is a generic term for a set of well-known techniques that have a long history in bibliometric studies of scholarly communication. [6] As artifacts of the scholarly communication process, citations can reveal formal communication patterns. Methods of citation analysis are unobtrusive and can be highly reliable.

Citation-based measures are not the only, or even necessarily the best, measures of such impact. There are other approaches to evaluating the impact of e-journals. Publishers can gain some information regarding use by noting the number of subscriptions to their journals or by counting the number of times articles are accessed or downloaded from host servers. These data provide useful indicators of one type of use, but they fail to assess the extent to which e-journal articles play a role in scholarship, research, and the advancement of knowledge.

Citation analysis assumes that references to a particular journal article reflect a scholarly impact of that article on the author of the citing work. It further assumes that the accumulated total of citations to a given author's works in some sense reflects the impact of that author on scholarship and research.

The same assumption applies to the assessment of the impact of journals--that the accumulated total of citations to all articles published by a journal is an indicator of the impact of that journal on the relevant discipline(s). Many studies have used citation-based measures as evaluation instruments for journals, research performance, university departments, universities, and published works. [7]

The main defect of citation analysis lies in the extent to which the primary assumption underlying all approaches to citation indexing--that citation reflects scholarly impact--is false. While the meaning of citation has been debated, [8] citation analysis is generally regarded as a valuable tool for determining the impact of scholarly works.

Citation analyses are normally conducted on the references of thousands of print (and a few electronic) journals. The Institute of Scientific Information (ISI) publishes three citation indexes--Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI)--that allow searches of journal names as cited works.

When a cited work search on an e-journal such as The Public-Access Computer Systems Review (PACS Review) is conducted on SSCI, one retrieves bibliographic citations to all the articles in journals indexed by SSCI that have cited a PACS Review article in one or more reference.

ISI represents the thousands of scholarly journals it indexes as the most significant journals in the various fields and disciplines of scholarship. Thus, a cited reference search should retrieve the majority of all citations made in scholarly journals to the cited work by the most important scholarly papers. It will not reveal citations to a work made in less significant journals or in books, dissertations, or other materials. In the context of the World-Wide Web, a cited work search will not reveal the hypertext links (analogous to citations) to works that are made in Web pages. However, these limitations notwithstanding, the ISI indexes are widely recognized as providing a valuable measure of the formal scholarly impact of articles, journals, and authors.

3.0 Methods

The following steps were taken to select a set of e-journals for study and to generate citation statistics for them:

  1. Define the sample of scholarly, peer-reviewed e-journals.

  2. Check for each of the e-journals in Journal Citation Reports (JCR).

  3. Search for journals not in JCR in the cited work field of SciSearch, Social SciSearch, and Arts & Humanities Search, being sure to identify all possible forms of the name of the cited journal.

  4. Print and verify citations.

  5. Perform analyses on the citation data.

3.1 Sample

All known scholarly, peer-reviewed e-journals with a reasonable chance of being cited were selected for study.

The time lag in the publication schedules of print journals is substantial, and many months are needed to provide time for a journal article to be read, to become part of an author's thought, and to be cited in a research paper. For these reasons, the study was limited to those scholarly, peer-reviewed e-journals that began electronic publication no later than 1993 (many well prior to 1993). There were thirty-nine such e-journals. Appendix A lists the e-journals in the sample and the year in which they began electronic publication. The journals were drawn from two directories of e-journals; detailed procedures for selecting the sample are reported elsewhere. [9]

3.2 Journal Citation Reports (JCR)

JCR is a tool published by the Institute for Scientific Information that includes several citation-based measures of journal impact for the source journals indexed. These measures will be discussed later. The first step in the present research was to check in JCR to see if the e-journals in the sample were included; only the Bulletin of the American Mathematical Society was there. To perform citation-based analyses on the remaining 38 e-journals, it was necessary to conduct cited work searches on the three ISI databases: SciSearch, Social SciSearch, and Arts & Humanities Search.

3.3 Cited Work Searches and Form of Name Identification

All cited work searches were done on the DIALOG system. DIALOG's OneSearch was used to search all three of the ISI citation databases simultaneously. This made it unnecessary to try to predict which database(s) should be searched for each journal (a problem for e-journals with strong interdisciplinary connections). (It is conceivable, for example, that the e-journal Psycoloquy might be cited by an article in a science journal or even by a humanities journal.)

The cited work field, as found in the ISI databases loaded on DIALOG, is a phrase-indexed field of at most 23 characters, the first three of which are "CW=." Thus, there are only 20 characters in which to represent the title of a given journal (or book). Finding all of the forms in which a title like Psyche: An Interdisciplinary Journal of Research on Consciousness appears in the databases is the most challenging part of doing a cited work search. For Psycoloquy, seven different forms were found (see Table 1). Note the several abbreviations and misspellings.

Table 1. Forms of Name for Psycoloquy in ISI Databases

For phrase-indexed fields such as cited work, a single entry is made in the index for each cited work. That entry is in the form of a long character string, or phrase. The phrase, or at least the first part of it, must be typed exactly in order to retrieve records. What is retrieved in such a search are bibliographic records of journal articles that have cited the work in question one or more times. The cited reference, in an abbreviated form that includes the cited author's name and other information, can then be printed.

To conduct a cited work search on DIALOG, one must first EXPAND on reasonable character strings, and then SELECT apparent hits. The EXPAND command opens a window to a section of the alphabetically arranged index entries, from which likely entries can be spotted and selected.

For example, consider the Journal of Statistics Education, a relatively simple title for which to search. The following considerations are relevant to one's strategy.

The procedures followed in the ISI databases for representing journal names are inconsistent. In particular:

The result of using the EXPAND command on the character string "CW=J STAT" is given in Table 2. Twelve index entries alphabetically related to "J STAT" are displayed in response to the command.

Table 2. EXPAND Cited Work Entries for "J STAT"

Ref   Items  Index-term
E3        2 *CW=J STAT
E4        1  CW=J STAT ANAL
E5       13  CW=J STAT ASS
E6        1  CW=J STAT ASS SEP
E7        5  CW=J STAT COMP
E8        1  CW=J STAT COMP S
E9        6  CW=J STAT COMP SIM
E10       1  CW=J STAT COMP SIMU

          Enter P or PAGE for more
? P

Ref   Items  Index-term
E15       1  CW=J STAT COMPUT
E24       1  CW=J STAT HPYS

The PAGE command was used to display the next twelve entries. If it were used to index the name of this e-journal, the phrase "J STAT E . . ." would fall between the entries E23 and E24. Thus, this particular character string did not generate any hits. There are, however, several forms of the name of what is evidently the Journal of Statistical Computation and Simulation.

EXPAND commands were given for several forms of this e-journal name, and the results were carefully reviewed for instances that might represent the Journal of Statistics Education (see Table 3).

Table 3. Forms of Name for the Journal of Statistics Education Checked with EXPAND Command
     CW=J OF ST
     CW=J STAT

Only two citations to this e-journal were found, both under the entry "CW=J STATISTICS ED."

This example demonstrates the problematic nature of conducting cited work searches on the ISI databases. A "good" search depends on trying all reasonable combinations of words and abbreviations. Errors are possible, even inevitable, despite the best efforts of the searcher. Thus, there is no guarantee that the results of the cited work searches to be presented here are complete. However, as the example demonstrates, a great deal of effort was expended in trying many combinations for each e-journal.

When all forms of a given e-journal were located, they were combined into a single set. The RD (remove duplicates) command was then used, because the searches were conducted in three ISI databases simultaneously and some citing journals are indexed by more than one ISI database. Using DIALOG's KWIC feature, bibliographic citations for the citing articles were printed, as well as the portion of the bibliography that included the cited reference(s). This was often necessary to verify that the cited journal was in fact the journal that was sought and to identify the cited author(s) and year(s).

A "citation" is interpreted by ISI as a unique reference to a work in a citing article. A given citing article (called a "source article" by ISI) may include several different references to articles in the same journal. Each of these is counted as a separate citation. Thus, the number of articles citing a work must be less than or equal to the number of citations to the work. Typing the citing articles in the KWIC format reveals all the citations. The number of citations can also be counted directly, without using KWIC, by using the DIALOG command SET POSTINGS ON and displaying the relevant retrieval set. [10]

Besides the form-of-name difficulty, there were other problems with some of the e-journals. Some e-journals with print counterparts were publishing in print form before they became electronic, and they are currently publishing two parallel versions. In order to eliminate citations to articles published only in print form, this study excluded citation data from these journals prior to the start of parallel print and electronic publication.

Another problem with a few e-journals was the difficulty in separating out the e-journal citations from citations to other books or journals of the same name. The worst example of this problem was Psyche. Psyche is the name of at least two other journals and many books. A cited work search on "CW=PSYCHE" retrieved more than 5,000 citing articles. If the publication dates were limited to the years in which the e-journal Psyche was published, there were still more than 600 citing articles. Without a great deal of additional work and significantly greater cost, including finding and inspecting the original citing articles in many cases, it was impossible to limit retrieval to the e-journal Psyche. This was not done in this instance, and the number of times the e-journal Psyche has been cited is unknown. The findings to be presented here do not include Psyche. The same problem was also experienced with a few other e-journals, but on a much smaller scale, and these ambiguities were resolved.

4.0 Analysis

The citation data reported here were gathered during February 1996. Eleven of the 39 e-journals for which cited work searches were conducted have print counterparts, and are thus published in "mixed," or parallel, formats, while the remaining 28 e-journals are "pure"--they are only published in electronic form. It is not possible from ISI's citation data (nor, indeed, from most bibliographic citations) to determine whether it is the print version or the electronic version of an e-journal article that was read and cited by an author.

Since the purpose of this study is to measure the impact of electronic journals, the data for these eleven e-journals must be treated with great caution. The chances are very good that the great majority of citations to most of these e-journals were generated by the print version of the journal rather than the electronic version. Although every scholar reads print journals, not every scholar reads electronic journals. Thus, for e-journals with a print counterpart, the chances are much higher that a given article was discovered, read, and cited as a result of its appearance in print than from its publication in electronic form.

4.1 Total Number of Citations

The total number of citations tends to measure the cumulative effect of a journal, since its articles can generate citations and credit for the journal for decades after publication. It is a long-term measure, clearly biased in favor of older journals that have been publishing for many years. Another characteristic of this measure is that, all other things being equal, it favors journals that publish many articles per year over those that publish only a few. The number of articles published in a given year will clearly affect the total number of citations received in subsequent years.

4.1.1 Frequency Distribution for Citation Data

Table 4 provides a frequency distribution of the citation data--the total number of citations to each e-journal in the sample.

Table 4. Frequency Distribution for Citations to E-Journals in the Sample*
Total Number of
Citations to E-Journal**
Number of E-Journals
0 15
1-5 13
6-10 3
11 1
... ...
14 1
... ...
35 1
... ...
38 1
... ...
111 1
... ...
190 1
... ...
> 1,500 1

*The number of citations to Psyche could not be determined.

**For e-journals with print counterparts, the number of citations includes all citations to articles published in the journal during the period the journal was published electronically.

With few exceptions, the e-journals in the sample have not been cited very often. Fifteen e-journals have not been cited at all, in their entire existence, and thirteen others were cited only between one and five times. The median of the distribution is one--fully half of the e-journals were cited either zero or one times. The distribution is highly skewed, with the great majority of e-journals being uncited, or very nearly so.

4.1.2 Top Eight E-Journals

A few e-journals are notable exceptions. Eight have been cited ten or more times. These e-journals are listed in Table 5, along with their disciplines or fields, whether or not the e-journal has a print counterpart, and the total number of times the journal has been cited. Three of the eight e-journals, and three of the top five, are also published in print form.

Among the top eight e-journals, three are from the sciences, five are from the social sciences, and none are from the humanities.

Table 5. The Eight Most Highly Cited E-Journals
Name of E-Journal Discipline Print Version? Number of Citations To the E-Journal*
Bulletin of the American Mathematical Society [11] Mathematics Yes > 1,500
Online Journal of Current Clinical Trials [12] Medicine No 190
The Public-Access Computer Systems Review [13] Library and information science (Yes) 111
Digital Technical Journal [14] Computer science (No) 38
Psycoloquy [15] Psychology No 35
Interpersonal Computing and Technology [16] Effects of technology on society No 14
Electronic Journal of Communication [17] Communication No 11
Postmodern Culture [18] Modern culture No 10

* For the Bulletin of the American Mathematical Society, data are for 1994 only. Data for all other journals include the total of all citations to the journal since publication began in electronic form to the present.

The top five e-journals are the Bulletin of the American Mathematical Society (BAMS), the Online Journal of Current Clinical Trials (OJCCT), The Public-Access Computer Systems Review (PACS Review), the Digital Technical Journal, and Psycoloquy.

4.1.3 Comparative Baseline Citation Data

The number of citations to journals in a discipline varies widely from field to field, a point made by Eugene Garfield [21] and since demonstrated in other studies. [22] Thus, it is unfair and misleading to compare the citation productivity of authors and journals in different fields (e.g., to draw inferences about quality or impact).

To put the numbers in Table 5 in perspective, we must make comparisons to print journals in the same fields. Another Institute for Scientific Information product, Journal Citation Reports (JCR), is available for such a comparison. JCR compiles and reports several citation statistics for the top journals in all fields.

As noted earlier, among the 39 e-journals in the sample, only the Bulletin of the American Mathematical Society appeared in the Journal Citation Reports lists, which necessitated the DIALOG searches on the three ISI databases reported earlier. Some of the e-journals in the sample--especially the eight listed in Table 5--are seemingly candidates for addition to Journal Citation Reports.

Table 6 reports JCR data for six fields in which some of the most highly cited e-journals are published: communication, computer science, information and library science, mathematics, medicine, and psychology. The JCR on CD-ROM (Version 1.0 for Windows; published in 1995) was used to produce these data. The JCR is in two parts, one part for the social sciences (1,402 journals) and the other for the sciences (4,513 journals). The "filter" feature was used to create a category of journals in each of the six fields. The journals in each category were then ranked by total number of citations received in 1994 (the last year for which JCR citation data were available) and printed. It was then easy to determine the top-and bottom-ranked journals, the three quartiles, and other percentiles.

As would be expected from the fact that the numbers of journals and researchers in each field vary considerably, Table 6 shows that the number of citations received by the median journal in each discipline differs widely from field to field. Citation peculiarities and practices of the fields are probably different as well. The median journal in library and information science, for example, received only 54 citations in 1994. The factors by which library and information science journals must be multiplied to make the median journals equivalent in terms of total number of citations received are: communication (3.6), computer science (3.2), mathematics (7.0), medicine (8.5), and psychology (4.4). The other quartiles also differ significantly from field to field.

Table 6. Number of Citations Received in 1994 by Journals in Several Fields
Name of Field and Number of Journals in Ranking (N) First Quartile Journal Median Journal Third Quartile Journal
Communication (N=23) 381 194 132
Computer science (N=205) 538 172 65
Library and information science (N=59) 136 54 30
Mathematics (N=202) 751 379 197
Medicine (N=114) 1,632 461 202
Psychology (N=99) 717 240 96

The citation counts in Tables 5 and 6 are not directly comparable, for several reasons:

Consequently, Table 6 is offered only to give the reader a general sense of how often journals in various fields are cited and to provide a very rough yardstick against which the impact of the e-journals in the sample can be measured. Having said this, if one compares the citation counts in Table 5 with those in Table 6, it is clear that most e-journals are having very little impact on formal scientific and scholarly communication.

4.2 Impact Factor

For these reasons, ISI created another citation-based measure of effectiveness, the impact factor, which normalizes the citations received by journals and looks only at the previous two years of publication.

The potential impact on scholarship and research is clearly a function of the number of articles published. If a journal publishes only a few articles each year, it cannot influence many other researchers and writers, and it will not be cited very often. The long-term impact of a journal can be assessed by the total number of citations received overall, or in a given year. The impact factor, somewhat misnamed, is defined by ISI as an average. Table 7 shows the formula for computing the 1995 impact factor for a given journal.

Table 7. 1995 Impact Factor for a Given Journal

     Impact factor for 1995 =

     (Number of citations received in 1995 to 1993 and 1994
        (Number of articles published in 1993 and 1994)

The impact factor reflects the impact of an average article published in the journal; it normalizes the number of citations, and it does not discriminate among journals according to the number of articles published in a given year or their longevity. It measures not so much the impact of a journal, but the impact of the typical article appearing in the journal. A journal can have a high impact factor, but because it publishes only a few articles each year or it has just commenced publication, it may have a very small overall influence on the scholarly community.

4.3 Immediacy Index

The immediacy index measures the extent to which articles make a quick impact on readers--the timeliness or currency of the journal. Historical journals would presumably have low immediacy indexes; cutting edge medical journals would have relatively high immediacy indexes. One would expect e-journals to have high immediacy indexes, since speed of publication is one of the most often cited advantages of e-journals. Table 8 shows the formula for computing the 1995 immediacy index for a given journal.

Table 8. 1995 Immediacy Index for a Given Journal
     Immediacy index for 1995 =

     (Number of citations received in 1995 to 1995 articles)
                    (Number of 1995 articles)

4.4 In-Depth Evaluation of Three E-Journals

Among the eight most highly cited e-journals identified in Table 5, the Bulletin of the American Mathematical Society and the Digital Technical Journal both had parallel print and electronic versions for most or all of their electronic history, resulting in the previously discussed problem of sorting out which citations are to the electronic versions. The last three e-journals listed in Table 5--Interpersonal Computing and Technology, the Electronic Journal of Communication, and Postmodern Culture--had far fewer citations than the others. These five e-journals were not studied any further.

Three e-journals were chosen for more detailed study: the Online Journal of Current Clinical Trials, Psycoloquy, and The Public-Access Computer Systems Review. Using the effectiveness measures presented in Table 9, 1994 JCR data for medicine, psychology, and library and information science journals were compared to 1995 data for OJCCT, Psycoloquy, and the PACS Review, respectively.

Table 9. Selected Measures of Journal Effectiveness
Measure Journal Characteristic Measured
Total number of citations received in a given year Long-term, aggregate impact of a journal in the year that the measurement is taken.
Impact factor Roughly, the average number of times an article is cited in the two years following publication.
Immediacy index Extent to which articles are cited quickly.
Number of articles published in a given year Potential impact of a journal in that year.

4.4.1 Online Journal of Current Clinical Trials

Table 10 compares 1994 JCR citation data for 114 medical journals (JCR category "Medicine, General & Internal") with 1995 citation data for the Online Journal of Current Clinical Trials. The measures reported are total citations, impact factor, immediacy index, and total number of articles published. For each measure, the data for the first-ranked journal, the median journal, the last-ranked journal, and data from OJCCT are reported. All but the OJCCT data are taken from the 1995 edition of Journal Citation Reports on CD-ROM.

The final column in Table 10 reports the percentile rank of OJCCT when compared to the 114 medical journals indexed by Journal Citation Reports. In total citations, for example, the top ranked journal (The New England Journal of Medicine) was cited nearly 100,000 times in 1994. With 82 citations, OJCCT is well below the median medical journal, and it ranks at the 9th percentile.

On the other hand, the percentile rankings for both impact factor and immediacy index for OJCCT are comparatively high, at 88 and 75 percent respectively. Thus, the average article in OJCCT is cited more than 88 percent of the e-journals indexed by JCR. The reason for the relatively small total number of citations received is that OJCCT has not published very many articles--only four in 1995.

Table 10. Publication and Citation Data for the Online Journal of Current Clinical Trials Compared to Journal Citation Reports Data for "Medicine, General & Internal" (N=114)
Measure First-Ranked Journal (1994)* Median Journal (1994) Last-Ranked Journal (1994) OJCCT (1995) OJCCT % Rank**
Total citations 98,940 461 4 82 9
Impact factor 22.7 .368 .01 2.54 88
Immediacy index 11 .247 0 .25 75
Total articles 785 107 0 4 7

* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, The New England Journal of Medicine, The New England Journal of Medicine, WHO Technical Report Series, and the British Medical Journal.

** Percentiles were computed by comparing 1995 OJCCT data to JCR data for 114 journals in medicine in 1994, the last year these data were available.

4.4.2 Psycoloquy

The results for Psycoloquy shown in Table 11 are quite similar to those of OJCCT--low percentile rankings for total citations and total articles, but a relatively high impact factor. While the average article published in Psycoloquy was cited more than twice as often as the median psychology journal, it published far fewer articles than the typical journal. Thus, its overall impact, as measured by total number of citations, is relatively small. The immediacy index for Psycoloquy was 0.0--no 1995 article was cited by an ISI-indexed journal in 1995. However, Psycoloquy's scholarly skywriting represents a rapid scholarly reaction that is different from, but in some sense equivalent to, citation.

Table 11. Publication and Citation Data for Psycoloquy Compared to Journal Citation Reports Data for "Psychology" (N=99)
Measure First-Ranked Journal (1994)* Median Journal (1994) Last-Ranked Journal (1994) Psycoloquy (1995) Psycoloquy % Rank**
Total citations 9,632 240 2 27 7
Impact factor 7.19 .415 .029 .94 76
Immediacy index 1.5 .083 0 0 27***
Total articles 510 30 0 4 7

* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, Psychological Bulletin, Psychological Review, Behavioral and Brain Sciences, and Psychological Reports.

** Percentiles were computed by comparing 1995 Psycoloquy data to JCR data for 99 journals in psychology in 1994, the last year these data were available.

*** Tied for last with 23 other journals.

4.4.3 The Public-Access Computer Systems Review

Table 12 examines The Public-Access Computer Systems Review. Compared to other journals in its field, the PACS Review enjoys the highest percentile rankings of all the e-journals studied. It was cited nearly as often in 1995 as the median journal in library and information science. It also had a high impact factor and a very high immediacy index score. These rankings were achieved in spite of the fact that the total number of articles published in 1995 was only at the 15th percentile. On the basis of these comparisons, it can be argued that the PACS Review is the most successful of all the e-journals examined in this study.

Table 12. Publication and Citation Data for The Public-Access Computer Systems Review Compared to Journal Citation Reports Data for "Information Science & Library Science" (N=59)
Measure First-Ranked Journal (1994)* Median Journal (1994) Last-Ranked Journal (1994) PACS Review (1995) PACS Review % Rank**
Total citations 725 54 1 39 37
Impact factor 1.314 .263 0 .43 71
Immediacy index 1.143 .021 0 .2 86
Total articles 109 22 0 5 15

* Journals ranked first by total citations, impact factor, immediacy index, and total articles were, respectively, the Journal of the American Society for Information Science, College and Research Libraries, the Journal of the American Medical Informatics Association, and Library Journal.

** Percentiles were computed by comparing 1995 PACS Review data to JCR data for 59 journals in library and information science in 1994, the last year these data were available.

4.5 E-Journal "Citation Classics"

Using the citation data already presented, it was relatively easy to identify the most highly cited e-journal articles. All e-journals except BAMS were included in this analysis. The most highly cited e-journal articles are listed in Table 13. All are from just two journals, OJCCT and the PACS Review.

The most highly cited e-journal article is by J. M. Elwood, "The Effectiveness of Breast Cancer Screening by Mammography in Younger Women," which was published in OJCCT. It had earned 59 citations as of February 1, 1996. It is worth noting that the Elwood article has been cited more often than all but three of the e-journals studied. In fact, each of the seven articles has been cited more often than 33 of the 39 e-journals studied. The seven articles account for 151 of the 444 citations received by all the e-journals together (again excluding BAMS).

Table 13. The Seven Most Highly Cited E-Journal Articles
No. of Citations (2/1/96) Article
59 Elwood, J. M., B. Cox, and A. K. Richardson. "The Effectiveness of Breast Cancer Screening by Mammography in Younger Women." Online Journal of Current Clinical Trials, 25 February 1993, doc. no. 32.
26 Villari, P., C. Spino, T. C. Chalmers, J. Lau, and H. S. Sacks. "Cesarean Section to Reduce Perinatal Transmission of Human Immunodeficiency Virus. A Metaanalysis." Online Journal of Current Clinical Trials, 8 July 1993, doc. no. 74.
14 Dickersin, K., and Y. I. Min. "NIH Clinical Trials and Publication Bias." Online Journal of Current Clinical Trials, 28 April 1993, doc. no. 50.
14 Harnad, Stevan. "Post-Gutenberg Galaxy: The Fourth Revolution in the Means of Production of Knowledge." The Public-Access Computer Systems Review 2, no. 1 (1991): 39-53. [23]
14 Klein, M. C., R. J. Gauthier, S. H. Jorgensen, J. M. Robbins, J. Kaczorowski, B. Johnson, M. Corriveau, R. Westreich, K. Waghorn, M. M. Gelfand et al. "Does Episiotomy Prevent Perineal Trauma and Pelvic Floor Relaxation?" Online Journal of Current Clinical Trials, 1 July 1992, doc. no. 10.
12 Okerson, Ann. "The Electronic Journal: What, Whence, and When?" The Public-Access Computer Systems Review 2, no. 1 (1991): 5-24. [24]
12 Wiggins, Rich. "The University of Minnesota's Internet Gopher System: A Tool for Accessing Network-Based Electronic Information." The Public-Access Computer Systems Review 4, no. 2 (1993): 4-60. [25]

5.0 Summary and Conclusions

The citation data reported in this study show that the great majority of scholarly, peer-reviewed e-journals have had essentially no impact on scholarly communication in their respective fields. Only eight of the 39 e-journals studied have been cited ten or more times over their lifetimes. Given that eleven of the e-journals have print counterparts, these findings are especially telling. The top eight e-journals are from the sciences, medicine, and social sciences.

To be meaningful, comparisons between journals must be for journals in the same field, since disciplines vary widely among themselves in their citation practices. Using Journal Citation Reports data, the top three e-journals--OJCCT, the PACS Review, and Psycoloquy--were compared to other journals in medicine, library and information science, and psychology, respectively. All three e-journals ranked highly in impact factor, which measures the impact of the typical article during the two year period following publication. OJCCT and the PACS Review also ranked well above the median journals in their fields on the immediacy index, which is a measure of the speed with which published articles are cited. The PACS Review also earned a respectable 37th percentile ranking in total citations. All three e-journals ranked poorly in total number of articles published.

Based on these findings, I conclude that the Online Journal of Current Clinical Trials, The Public-Access Computer Systems Review, and Psycoloquy all show great promise of becoming one of the top journals in their respective fields. However, all are publishing far fewer articles in a given year than most other journals in their disciplines. Thus, while their high impact factors for a typical article show great promise, the overall scholarly impact of these e-journals on their disciplines is not great. Indeed, they cannot have a major impact until they publish many more articles annually than they presently do, while maintaining the present overall high quality of their articles. More authors will need to view e-journals as legitimate publication vehicles before e-journals can assume a significant role in the scholarly communication process.


1. For example, see: Andrew M. Odlyzko, "Tragic Loss or Good Riddance? The Impending Demise of Traditional Scholarly Journals," International Journal of Human-Computer Studies 42 (1995): 71-122.

2. Quoted in: M. P. Collins and Z. L. Berge, "IPCT Journal: A Case Study of an Electronic Journal on the Internet," Journal of the American Society for Information Science 45 (1994): 771-776.

3. I am grateful to the Online Computer Library Center, Inc. (OCLC) for partial funding of the overall study.

4. Stephen P. Harter and Hak Joon Kim, "Electronic Journals and Scholarly Communication: A Citation and Reference Study," in Proceedings of the ASIS Midyear Meeting (San Diego: American Society for Information Science, 1996), 299-315. Also available at <URL:http://ezinfo.ucs.indiana.edu/~harter/harter-asis96midyear.html>.

5. Stephen P. Harter and Hak Joon Kim, "Accessing Electronic Journals and Other Epublications: An Empirical Study," College & Research Libraries 57 (September, 1996), in press.

6. As excellent collection of such papers that includes an outstanding bibliography of additional sources is given in: Christine L. Borgman, ed., Scholarly Communication and Bibliometrics (Newbury Park, CA: Sage Publications, 1990).

7. For an early treatment of these applications, see: Francis Narin, Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity (Cherry Hill, NJ: Computer Horizons, Inc., 1976).

8. For example, see: T. A. Brooks, "Evidence of Complex Citer Motivations," Journal of the American Society for Information Science 37 (1986): 34-36; S. E. Cozzens, "What Do Citations Count? The Rhetoric-First Model," Scientometrics 15 (1989): 437-447; G. N. Gilbert, "Referencing as Persuasion," Social Studies of Science 7 (1977): 113-122; M. H. MacRoberts and B. R. MacRoberts, "The Negational Reference: Or the Art of Dissembling," Social Studies of Science 14 (1984): 91-94; and H. G. Small, "Cited Documents as Concept Symbols," Social Studies of Science 8 (1978): 327-340.

9. Harter and Kim, "Electronic Journals and Scholarly Communication."

10. I thank Peter Ingwersen for bringing my attention to the SET POSTINGS ON command for counting citations. This and other methods of conducting online citation analyses are described in: Finn Hjortgaard Christensen and Peter Ingwersen, "Online Citation Analysis: A Methodological Approach," Scientometrics 37 (September, 1996), in press.

11. See <URL:http://www.ams.org/journals/bull/>.

12. See <URL: http://www.oclc.org/oclc/promo/ejo_list.htm#cct>.

13. See <URL:http://info.lib.uh.edu/pacsrev.html>.

14. See <URL:http://www.digital.com/info/DTJ/home.html>.

15. See <URL:http://cogsci.ecs.soton.ac.uk/~harnad/psyc.html>.

16. See <URL:http://www.helsinki.fi/science/optek/>.

17. See <URL:http://cios.llc.rpi.edu/www/ejcmain.htm>.

18. See <URL:http://jefferson.village.virginia.edu/pmc/contents.all.html>.

19. The 1990 volume was published in book form in 1992; the 1991 volume in 1992; the 1992 volume in 1993; the 1993 volume in 1994; and the 1994 volume in 1996.

20. Stevan Harnad, "Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry," Psychological Science 1 (November 1990): 342-344. Also available at: <URL:ftp://cogsci.ecs.soton.ac.uk/pub/harnad/Harnad/harnad90.skywriting>.

21. Eugene E. Garfield, Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (New York: Wiley, 1979).

22. P. Vinkler, "Possible Causes of Differences in Information Impact of Journals from Different Subfields," Scientometrics 20 (1991): 145-161.

23. See <URL:http://info.lib.uh.edu/pr/v2/n1/harnad.2n1>.

24. See <URL:http://info.lib.uh.edu/pr/v2/n1/okerson.2n1>.

25. See <URL:http://info.lib.uh.edu/pr/v4/n2/wiggins1.4n2> and <URL:http://info.lib.uh.edu/pr/v4/n2/wiggins2.4n2>.

Appendix A. The 39 E-Journals in the Study

The year that the journal began electronic publication is shown in parentheses after the journal's name.

About the Author

Stephen P. Harter, Professor, School of Library and Information Science, Indiana University, Bloomington, IN 47405. Internet: harter@indiana.edu.

About the Journal

The World-Wide Web home page for The Public-Access Computer Systems Review provides detailed information about the journal and access to all article files: <URL:http://info.lib.uh.edu/pacsrev.html>.


This article is Copyright © 1996 by Stephen P. Harter. All Rights Reserved.

The Public-Access Computer Systems Review is Copyright © 1996 by the University Libraries, University of Houston. All Rights Reserved.

Copying is permitted for noncommercial, educational use by academic computer centers, individual scholars, and libraries. This message must appear on all copied material. All commercial use requires permission.