Citation Analysis As A Tool In Journal Evaluation

to have made more use of citation analysis in studies of science policy thanhave any other scientists ( 12–14 ).One must remember that the listingswere prepared from a 3-month sampleof journal issues. The size of the sampleis certainly more than adequate for statisticalpurposes,and, as noted, thesample has been matched against another sample of more than adequatesize. It is nevertheless possible that random events in journal publishing haveintroducedsome degree of distortion,For example, a journal may hirvc, in c time period covcrcd by the sample,departedfrom its usual policy of accepting only original research communicationsand publishedone or tworeview articles wltb extensivehibliog.raphics.In the sample, that journalwould appear to cite other publicationsSOURCE JOURNALREFERENCE JOURNALTOTAL19691968more wtdely and more frequently thanit actually does on the average. Or. ajournal may have published an articlethat has since been cited with extraordinary frequency (/5).In such a case, asingle article will have had an inordinate influence on the r nking of the)ournal( 16). Finally, a journal thatpublishesrelativelyfew articles,butarticles of high quality that both citeand arc cited frequently,may seem tohave considerablyless impact than itactually does, particularlyif the journalappears infrequentlyor irregularly andthus cscapcs representativeinclusion ina sample of this type.In antilysis of the sample, an immensely irksome problem was the inconsistency with which different authorsand editors abbreviate journal titles intheir rcfcrcnces. As far as possible, thisNUMBEROF TIM ES CITEO1967 19s43 1965 1964 19631962 19611960 RESTJ LIBR AUT‘ 168* ‘ 15 50 35 19 8 14 5 6 3 1 12PROGRAM1536510000000J LIBRARY AUTOMAT ION---- 7 -. -.3. -.4. .-.0 -----0.- .0-. ---0.---0-.-.0----.0 --.0 -----O1469ALL OTHER (120)403018814563112J LIPIO RES ‘ 313*J BIOL CtiEM43J LIPID RES -.-. -.-.-.---. -.2SBIOCHtM BIOPHYS ACTA19BIOCHEM J . . . . . . 13J AMER CHEM SOC12BIOCHEMISTRY . . .- 9J CHEM SOC LONDON5J CHROMATOGR.5METHODS ENZYMOL5ALL OTHER (89}----------------- 124‘ 15 36 35 26 25 29 20 10 15 15 87263613101119.-.-- 5. -.5--.-4 .-.-2. -.4 ------ 1 .-.2 .0------ 3.----. 1.-113132310221.-- 1 -.--. 1.-.- 3.-. --0 .0------ 0-.--. 1 .-.2 ------ 1 .-. 1-----.300000000loll.--.-.0. -----3.-. 0.-. -.2-.--. 1.-.-2 -.-.1 -.-.0 .-.-. 0. ---0 .--.--000100000103.--.0 -. -.1.-. 1 .--.-.1 -.-.-0.-.- 1 -. -.0. -. 1.-.---0 -.0. -.--000000001004-.--.5 . 12 .19---- 10.-.8.-.16-.--10-.-.2 --.--.5 .-.6 .-31J LOND MATH 743” 21 44 58 52 42 34 31 38 29 33 361J LOND MATH125543021311526P LONDON MATH SOC -.37 -. 4.- .5.-.-.7 .-.- 1 --.-.2 .2 .-. -0. --.1 --.--1 .-.0 .-.14Fig. 3. Statistics on citing journals. The data on each citing journal (“source journal”)show the total number of references each journal contained in the last quarter of1969 and the distribution of [hat total by publication date of journal issues referredalljournals(“referencejournal”’)to. For each citing journal, the figure identifiesor more during the quarter year (and gives the distribution by pubcitedfive timeslication data of cited issue). Journals cited less than five times are grouped as “allother.” The data are taken from a list of journals processed for the SCl during thelast quarter of t 969.530

side a regularly enumerated series mustbe accounted for. In a few cases, journals periodically change their titles ll variants of the sametitles and their abbreviations.Some ideaof the work involved in this standardiza-tion can be had from the fact that therewere more than 100,000 different abbreviations for the 12,000 individualjournal titles cited in the -month sample ( /7). Inconsistency was made worseby inaccuracy. In some cases, it waspossible 10 disentangle the results ofbibliographicample, whenhave beenident ify Solidcarelessness—as,for exSol. St. Phys. proved toused indiscriminatelyto(AcademicState PhysicsPress ): Solid Stute Physics,o (aof FiZikaand even PhysiraProceedingsLondon;StutusSo victcover-to-coverTverdogoSolidiTela );( Akade-mic-Verlag). In other cases, however, itwas impossible,withoutgoing to inordinate expense, to determineexactlywhich journal was being cited—for example, when Arm. Phys. was found tohave been used for Annalers der Physik,and Annales deAnnuls of Physics,Physique.It is not surprising that theeditors of at least one publishing house,having decided that the problem ofunique and unambiguous journal titleabbreviations is simply insoluble, nowuse full titles in every reference to ajournal.Finally, it was necessary to makesomearhi trary decisionsin order toavoid unduly complex bibliographicaltechnicalities. Journals merge; they splitinto new journals, or into “sections”that may be pubIished separately or together. They change titles, with or without continuing their numbering of volumes and issues. Some journals appearin one or more translations; some suchtranslations are complete, others selective, and some are similarly, othersdifferently, numbered. Supplements out-531single issues to note special subject matter (18). Serials librarianshipaboundsin difficulties of this type, and there isfrequently disagreement on how best tohandle them. Briefly: a journal published in sections is considered a singleof journals arejournal: translationsidentified with the original versions;changes of title have been ignored andprevious volumes attributed to the current title; journals absorbed or incorporatedby other journals have been credited to a new or remaining title; supplements have been considered as issues inthe regular series. For the purposes ofthis analysis, such arbitrary decisionsseem justified; as required in the besuchabetween changedtranslations,and soas to differentiatetitles,forth.sect lysis.It is a listing of journaltitlqranked bythe frequency with which they wereof journalsincited in the referencesdexed for SCL This partial listing givesthe top 152 of the 565 most frequentlycited journals of science and technology. (The top 152 accourrt for 50 percent of all references to journals. )It is apparent, even from the makeupof this partial listing, that a good multidisciplinaryjournalcollectionneedcontain no more than a few hundredtitles. That is not to say that larger collections cannot be justified, but it doessay something indisputable, in terms ofcost and benefit, about how large rs jour-

1WmNo.(11(2)12::J AM CBEM SOCPHYSREVJ BIOLCHEMNATuRELONDONJ CHEMSOCJ S;U; :TAP NATACADSC IW;; MJPHYSREvLETTCR ACAOSC IAM J PHYSIOLJ ORG CHEM”!.J APPL‘-” ExPBIOLMEOP SOcJ MOLi310LJ PHVSIOLLONOP ROYSOC LONOJ CELLEIOLJ CLININVESTJ PHVSCHEMCHEMBERNEW ENGLJ MEOJ AM MEfJ ASSBRITMEO JSOVPHYSJETPAS TROPHYSJANALYTCHEMJ BACTERIOLBIOCHEMISTRYN!JCLPHVSPH” , .!,,.!!,. TRAHEORONLETTJ’ExP MEOANNNY ACAOSC IARCHBIOCHEMBIOPHYSJ GE OPHYSRESt .0(.“. . . . . . . . .il’OCHEM 810 PHYSRESFEDPJ PHYST FARADAYSOCACTACR YSTALLOGRDOKLAKAONAUKSSSRJ PHARMACOLExPTHER Nf u::fM.,1INORG‘U .!.!CIfiSTATESovPtHYS[: L]DCIRCULATIENOOCR INO ILOGV :::oCHEMA SC ANDCIMENTOFRANCEB Soc . CHIM.vIROLIJbYCANCERRESCAN J CHEMHE, ” C“l M APTAHij NGz NATti R60RizAMA J MEO. AB !NMEOTETRAHEDRONEXPCELLRE sLIE EIGSANN 1 CHEMANN[1NT MEOPHILIMAGJ CLI! N EW:5LMETABJ APPLPHACr.0“” T510L.,.SCANOSOC JAPJ PHYS PHYSCIRCRESPHY1O’?AIHO[OGYJ 2254B244424312416237623.49228022a9;: 14234530 5bb125172b1247156121607839431’ 5245421202i009Fig. 4. The 152 most frequentlycited journalsrankedby frequencyColumn1 gives rank,of cilalionin journals covered by Ihe SCI.and column 2 gives abbreviationsof the titles of cited journals.Column3 shows the total number of limes each journal was citedduring the last quatier of 19s39. Column4 gives an estimate of thetotal number of citations in 1969 of items publishedin 1967 and532

Item80818283::8687888990:;v 01 48149150151152CnationshwrntiNo.(1)77;:1 assto4r3k3m*UT alla 1s68In1U7and 196aArtkk%{4](s)Clke(2)AM J OBI: T,: WO:YNECOLPLAVTPHYINDENGCl HEMANNSURGB CHEMSO’IC JbPEuRJ BIO ICHEMGENETICSBLOODP IEEEJ OPTSOC N:;;7RI( O:!EM.PHYsloLMEDARCHIINTERNAM HEARTJJ ExPPSVCHOLJ GEN MI CR OBIOLJ COMPPHYSIOLPSYCHJ PHYSCHEM50 LIOSCANCERAM J P-,.,’ ’,!JL--1.J PHYS(HEMENZYMOLISRG NUCLCHEMTRICS3YNECOLOBSTET?EC 0 PnvsSOC AIME.“.J PHARMACO1.,l,.?.-. ., ,,.,,.I J I?OENTGENOLfiA JASMEI J CA ROIOLJI HIS TO CHEMCYTOCHEMJ PEo [ATJ ACOUSTSOC AM A; :W ISSENSCHAFTENii1SPECTROC141MACTAZ ANORGALLGCHEMJ PERSONSOCPSVCMOLRAOIOLOGV:MlI afll HYSCHEMLEIPZIG‘3 MA TOGRJ LMRUSE YLERSzHOPPEJ UROLARCHPA THOLARCHSURGAM J 01SCH ILOACT4MEO5cAN0ANNPHYSICSSPR HARB SYMPCOLDJ ORGANOME1CHEMpFLu GEC’C ,.Pw(lPT 0-PECTROSCuSSR LINwSCHRCHEMINOLONOBER SUNSENPHYSCHEMBIOCHEMPHARMACOLPHYSIOLREvJ BONEJOINTSURGJ F6EuROPHVS[OLCR SOC BIOLRECTRAVCUIM. -. . . . . . 08709535183b18.901101’J50772;148 83:j:;48967Q54 7380692888333b714354521601968 (the estimate was ma& by quadrupling the 1969 citations of1967 and 1968 items in the 3-month sample). Column S shows thetotal number of items processed from each journal by the S(Hduring 1967 and 1968. Column 6 indicates the impact f tor (averagecitations per published itern) derived by dividing the num hers incolumn 4- hy those in column 5.533

na[ collection need be (or how small itcan be) if it is to provide effectivecoverage of the literature most used byresearch scientists.It is also immediately apparent thatthe majority of all references cite relatively few journals. Figure 5, whichplots the distribution of citations amongcited journals, shows that only 25 journals ( little more than 1 percent of SCIcoverage ) are cited in 24 percent of allreferences; that only 152 journals (thoselisted in Fig. 4) are cited in 50 percent of all references; that only 767journals are cited in 75 percent of allreferences; and that only 2000 or sojournals are cited in 85 percent of allreferences. In addition, the data fromwhich Fig. 5 was plotted show that only540 journals are cited 1000 or moretimes a year, and that only 968 journalsare cited even 400 times a year. Whenone considers that only 165 or so journals publish 400 or more papers a year,the impact of the average paper mustbe recognized as relatively slight. Infact, the average paper is cited only1.7 times a year (19).This analysis gives good reason forconcern about any increase in the number of scientific and technical journals.It is not merely that increased numbersof journals make coverage of the literature more difficult, but that so manyjournals now being published seem coplay only a marginal role, if any, in theeffective transfer of scientific information, If one accepts the contention (highly debatable, in my opinion) that thereare s,”scienthedatahere indicate that only 5 to6 percent of them are being cited, Ifsuch percentages seem unrealisticallylow, it may be because the estimate of50,000 to 100,000 scientific and technical journals (requiring indexing andpresentedabstracting for “total coverage” of theliterature ) is itself as unrealistic as information scientists have for some timesuspected it must be (20). Meaningfuldiscussion of this point—the best ofcurrent serials catalogs notwithstanding—is probably impossible in the absenceof agreement on what, quantitatively aswell as qualitatively, constitutes a “scientific journal.” At the very least, itmay be advisable to distinguish asjournals only those periodicals that publish, for example, 20 or more articlesa year.The predominance of a small groupof journals in the citation network hasbeen confirmed by a weekly scanningof SCI input to a system for selectivedisseminationof information(SDI)(21). In this SDI system, a newly published article can be retrieved on thebasis of journals cited in the article’sbibliography or footnotes. This retrievalcriterion is known in an SDI profife asa “cited-journal question. ” A retrievalprofile consisting of only 25 differentcited-journal questions will retrieve 50percent of all articles processed for theSCI every week. In other words, half ofallarticlespublishedof the 25 mostcitefrequentlyat leastonecited journalsat least once.It is also interesting that a smallgroup of journals is found ,0 be predominant when the literature is analyzed in other ways. Figure 6, for example, plots numbers of articles ptsblished against numbers of jouraals. Itshows that, of the 2200 journals covered by the SCI in 1969, about 500publishedabout70 percent of all articles published. As shown in Fig. 7, asmall group of 250 journals providedalmost half of the 3.85 million references processed for the SCI in 1969.The predominance of cores of journalsis ubiquitous.An analysis of whti534

Cited journalsFig. 5. Distributionof citations(No.)among cited journals. The curve shows that a rela-tively small core of 152 journals accounts for about hatf of afl citations and thatonly- 2000 or so journals account for about 84 percent of all citations.firstpublished reports of newly synthesized compounds indexed forjournalsCurrentAbstractsojChemistryandresult:of the 183 journals indexed by thispublication in 1969, 11 percent of thejournals accounted for 60 percent ofthe new compounds, 22 percent of thejournals for 79 percent of the compounds, 32 percent of the journals for89 percent of the compounds, and soon (22). Chemical Absrracts presentsan even more striking example of th predominance:about 8 percent of thejournals it covers publish more than75 percent of the items it abstracts (23).The data reported here demonstratethe predominance of a small group offndexjournalsChemicusin thegives a similarcitationnetwork.ford’s bibliographicallaw concerningthe concentration and dispersion of theliterature of individual disciplines andspecialties (2). Going beyond Bradford’s studies, 1 can say that a combination of the literature of individualdisciplines and specialties produces amultidisciplinary core for all of sciencecomprising no more than 1000 journals.The essentialmultidisciplinarycorecould, indeed, be made up of as fewas 500 journals if, for example. one isattempting to satisfy the needs of libraries in developing countries.Other ConsiderationsCitationIndeed,the evidencefrequencyreflectsa joLlrnal’sseems so conclusive that I value and the use made of it, but there[ can with confidence generalize Brad- I are undoubtedly highly useful journals535

Sourcejournalsindexedin1969(No.)Fig. 6. Distribution of the number of published items among the ar?proximalely 2200joumafs covered by the SCI in 1969. The curve shows that a relatively small coreof journals carried the majority of items published.that are not cited frequently.Scientistsread some journals for the same reasonpeople read newspapers—tokeep upwith what’s going on generally—andthey may rarely or never cite suchjournals in their published work (24).A popular review journal such as Scle?r/ific American or a news-oriented journal such as New’ Scientisf may rank relatively low on a times-cited list (infact, Scierrfi/ic American is 449th, whileNew Scien/i.s/ ranks well below 1000),but that does not mean that they aretherefore less important or less widelyused than journals that are cited morefrequently. It merely means that theyare written and read primarily for somepurpose other than the communicationof original research findings.Citation frequency is, of course, afunction of many variables besides scientific merit. Some of them are knownor can reasonably ‘be assumed: an author’s reputation,controversialityofsubject matter, circulation, availabilityand extent of library holdings, reprintdissemination,coverage by secondaryservices, priority in allocation of research funds, and others. It is extremelydifficult, even when possible, to clarifythe relations among such variables andtheir relative impact on citation frequency, One such variable is, however,fairly obvious. If every article has anequal likelihood of being cited, it shouldfollow that the more articles a journalpublishes, the more frequently the journal will be cited. For the most part,the data show that such is indeed thecase. Although many articles are never536

(25, 26), 1 journalsin termsverymostisrarelyfoundfrequentlycitednotareof articlesthatalsoamongmost propublished.Citationfrequencyof a journal isthus a function not only of the scientific significance of the material it pubbycitation),butlishes (3s reflectedalso of the amount of material it publishes.In view of the relation between sizeand citation frequency, it would seemdesirable to discount the effect of sizewhen using citation data to assess ajournal’s impotiance. We have attempted to do this by calculating a relativeimpact factor—that is, by dividing thenumber of times a journal has beencited by the number of articles it haspublished during some specific period ofThe journal impact factor willthus reflect an average citation rateper published article (27). However,the development of impact factors thatfairly relate the size of a journal during{he cited years to its current citationrate is a formidable challenge to statistical analysis. With the SC1 data base,it is easy to determine how frequentlya journal has been cited within a givenperiod of time, but it is much moredifficult to agree on

Citation Analysis as a Tool in Journal Evaluation Journals can be ranked by frequency and impact of citations for science policy studies. Eugene Garfield [NOTE: Tbe article reptintedbere wasreferenced in the eoay vbi.b begins m @g. 409 is Volume I, IIS ia. #dverte#t omission W6Sdiscovered too Me to iwctnde i