Research on Social Work Practice

 

Advanced Search

Journal Navigation

Journal Home

Subscriptions

Archive

Contact Us

Table of Contents

Free Access - Register Here

Click here for free access to the SAGE eReference platform!

Sign In to gain access to subscriptions and/or personal tools.
This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Holden, G.
Right arrow Articles by Onghena, P.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?
Research on Social Work Practice, Vol. 16, No. 6, 613-624 (2006)
DOI: 10.1177/1049731506292570

An Assessment of the Predictive Validity of Impact Factor Scores: Implications for Academic Employment Decisions in Social Work

Gary Holden

New York University

Gary Rosenberg

Mount Sinai School of Medicine

Kathleen Barker

City University of New York: Medgar Evers College

Patrick Onghena

Katholieke Universiteit Leuven

Objective: Bibliometrics is a method of examining scholarly communications. Concerns regarding the use of bibliometrics in general, and the impact factor score (IFS) in particular, have been discussed across disciplines including social work. Although there are frequent mentions in the literature of the IFS as an indicator of the impact or quality of scholars' work, little empirical work has been published regarding the validity of such use. Method: A proportionate, stratified, random sample, of n=323 articles was selected from 17 Web of Science listed social work journals published during the 1992 to 1994 period. Results: The relationship between journals'IFSs and the actual impact of articles published in those journals (predictive validity) was r= .41 (short term) and r = .42 (long term).Conclusion: The practice of using the IFS as a proxy indicator of article impact merits significant concern as well as further empirical investigation.

Key Words: scientific communication • decision making • personnel promotion • personnel selection • teacher tenure • educational personnel • occupational tenure • career development • higher education • citation analysis • impact factor scores • validity • faculty • hiring • reappointment • social work education

References

  • Abbasi, K. (2004). Let's dump impact factors. British Medical Journal, 329-329. Retrieved October 16, 2004, from http://bmj.bmjjournals.com/cgi/content/full/329/7471/0-h
  • Adam, D. (2002). Citation analysis: The counting house. Nature, 415, 726-729.[Medline] [Order article via Infotrieve]
  • Aronowitz, S. (2001). The knowledge factory: Dismantling the corporate university and creating true higher learning. Boston: Beacon.
  • Baker, D. R. (1990). Citation analysis: A methodological review. Social Work Research & Abstracts, 26, 3-10.
  • Baker, D. R. (1991). On-line bibliometric analysis for researchers and educators. Journal of Social Work Education, 27, 41-47.
  • Balaram, P. (2000). The impact of publication lists. Current Science, 78, 1177-1178.
  • Barker, K. (1995). Contingent work: Research issues and the lens of moral exclusion. In L. Tetrick & J. Barling (Eds.), Changing employment relations: Behavioral and social perspectives (pp. 31-60). Washington, DC: American Psychological Association.
  • Barker, K. (1998). Toiling for piece-rates and accumulating deficits: Contingent work in higher education. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition (pp. 195-220). Cornell, NY: Cornell University Press.
  • Barker, K. (2001). Standard and nonstandard employment in higher education: Findings from the 1990's. In Commission on Professionals in Science and Technology (Ed.), Changing career paths in science and engineering: Report of a CPST workshop (pp. 12-30). Washington, DC: AAAS, Commission on Professionals in Science and Technology. Retrieved April 15, 2006, from http://www.cpst.org/Changing.pdf
  • Barker, K. (2003). Contingent work in the United States. In Supplement to the 19th Edition of the Encyclopedia of Social Work (pp. 9-19). Washington, DC: National Association of Social Workers.
  • Barker, K., & Christensen, K. (1998a). Introduction: Controversy and challenges raised by contingent work arrangements. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition(pp. 1-18). Cornell, NY: Cornell University Press.
  • Barker, K., & Christensen, K. (1998b). Charting future research. In K. Barker & K. Christensen (Eds.), Contingent work: American employment relations in transition (pp. 306-320). Cornell, NY: Cornell University Press.
  • Barker, K., & Kohout, J. (2003). Contemporary employment in psychology and future trends. In M. J. Prinstein & M. Patterson (Eds.), The portable mentor: Expert guide to a successful career in psychology (pp. 309-332). New York: Kluwer Academic/Plenum.
  • Beck, U. (2000). The brave new world of work: The political economy of insecurity. Cambridge, UK: Cambridge University Press.
  • Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36, 3-72.
  • Braxton, J. M., & Del Favero, M. (2002). Evaluating scholarship performance: Traditional and emergent assessment templates. New Directions for Institutional Research, 114, 19-31.[CrossRef]
  • Burgan, M. (1998, March/April). What we talk about when we talk about tenure. Academe, 84(2), 2-2.
  • Campling, J. (1992). Social-policy digest 81. Journal of Social Policy, 21, 99-118.
  • Campling, J. (1993). Social-policy digest 85. Journal of Social Policy, 22, 99-118.
  • Christenson, J. A., & Sigelman, L. (1985). Accrediting knowledge: Journal stature and citation impact in social science. Social Science Quarterly, 66, 964-975.
  • Cnaan, R. A., Caputo, R. K., & Shmuely, Y. (1994). Senior faculty perceptions of social work journals. Journal of Social Work Education, 30, 185-199.
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
  • Cole, J., & Cole, S. (1971). Measuring the quality of sociological research: Problems in the use of the Science Citation Index. American Sociologist, 6, 23-29.
  • Cole, J. R. (2000). A short history of the use of citations as a measure of the impact of scientific and scholarly work. In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: A festschrift in honor of Eugene Garfield(pp. 281-300). Medford, NY: Information Today.
  • Colquhoun, D. (2003). Challenging the tyranny of impact factors. Nature, 423, 479-479.[Medline] [Order article via Infotrieve]
  • Dahwan, S. M., & Gupta, B. M. (2004). Comparative evaluation of Indian physics research: Impact factor vs. citations frequency. Current Science, 86, 1194-1195.
  • Decker, O., Beutel, M. E., & Brahler, E. (2004). Deep impact— Evaluation in the sciences. Sozial Und Praventivmedizin, 49, 10-14.[Medline] [Order article via Infotrieve]
  • Fassoulaki,A., Sarantopoulos, C., Papilas, K., Patris, K., & Melemeni,A. (2001). Academic anesthesiologists' views on the importance of the impact factor of scientific journals: A North American and European survey. Canadian Journal of Anesthesia, 48, 953-957.[Abstract/Free Full Text]
  • Fava, G. A., & Ottolini, F. (2000). Impact factors versus actual citations. Psychotherapy and Psychosomatics, 69, 285-286.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Finkelstein, M. J., Seal, R. K., & Schuster, J. H. (1998). The new academic generation: A profession in transformation. Baltimore, MD: Johns Hopkins University Press.
  • Frank, M. (2003). Impact factors: Arbiter of excellence. Journal of the American Medical Library Association, 91, 4-6.
  • Furr, A. L. (1995). The relative influence of social work journals: Impact factors vs. core influence. Journal of Social Work Education, 31, 38-45.
  • Garfield, E. (1983). How to use citation analysis for faculty evaluations, and when is it relevant? Part 2. Current Comments, 45, 5-14.
  • Garfield, E. (1996). How can impact factors be improved? British Medical Journal, 313, 411-413.[Free Full Text]
  • Garfield, E. (1997). Validation of citation analysis. Journal of the American Society for Information Science, 48, 962-964.[CrossRef]
  • Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161, 979-980.[Free Full Text]
  • Garfield, E. (2003). The meaning of the impact factor. International Journal of Clinical and Health Psychology, 3, 363-369.
  • Gastel, B. (2001). Assessing the impact of investigators' work: Beyond "impact" factors. Canadian Journal of Anesthesia, 48, 941-945.[Free Full Text]
  • Glanzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53, 171-193.[CrossRef]
  • Hansen, H. B., & Henriksen, J. H. (1997). How well does journal "impact" work in the assessment of papers on clinical physiology and nuclear medicine. Clinical Physiology, 17, 409-418.[CrossRef]
  • Holden, G., Rosenberg, G., & Barker, K. (2005a). Tracing thought through time and space: A selective review of bibliometrics in social work. Social Work in Health Care, 41(3/4), 1-34.[Medline] [Order article via Infotrieve]
  • Holden, G., Rosenberg, G., & Barker, K. (2005b). Shallow science or meta-cognitive insights: A few thoughts on bibliometrics. Social Work in Health Care, 41(3/4), 129-148.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Holden, G., Rosenberg, G., & Barker, K. (2005c). Bibliometrics: A potential decision making aid in hiring, reappointment, tenure and promotion decisions. Social Work in Health Care, 41(3/4), 67-92.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Howard, M. O., & Howard, D. A. (1992). Citation analysis of 541 articles published in drug and alcohol journals: 1984-1988. Journal of Studies on Alcohol, 53, 427-434.[ISI][Medline] [Order article via Infotrieve]
  • Hutcheson, P. (1997). The corrosion of tenure: A bibliography. NEA Higher Education Journal, Fall, 89-106.
  • ISI. (2003). Journal Citation Reports. Retrieved November 12, 2004, from http://www.isinet.com/
  • ISI. (2004). Journal Citation Reports. Retrieved November 12, 2004, from http://www.isinet.com/
  • Katz, J. S. (1999). Bibliometric indicators and the social sciences. Retrieved July 27, 2003, from http://www.sussex.ac.uk/Users/sylvank/pubs/ESRC.pdf
  • Kirk, S. A. (1984). Methodological issues in the comparative study of schools of social work. Journal of Social Service Research, 7(3), 59-73.
  • Kostoff, R. N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics, 43 (1), 27-43.[Medline] [Order article via Infotrieve]
  • Kostoff, R. N. (2002). Citation analysis of research performer quality. Scientometrics, 53, 49-71.
  • Kreuger, L. W. (1993). Should there be a moratorium on articles that rank schools of social work based on faculty publications? Yes! Journal of Social Work Education, 29, 240-246.
  • Kurmis, A. P. (2003). Understanding the limitations of the journal impact factor. Journal of Bone and Joint Surgery, 85A, 2449-2454.
  • Linardi, P. M., Coelho, P. M. Z., & Costa, H. M. A. (1996). The "impact factor" as a criterion for the quality of scientific production is a relative, not absolute, measure. Brazilian Journal of Medical and Biological Research, 29, 555-561.[ISI][Medline] [Order article via Infotrieve]
  • Lindsey, D. (1976). Distinction, achievement and editorial board membership. American Psychologist, 31, 799-804.
  • Lindsey, D. (1978). The scientific publication system in social science. San Francisco: Jossey-Bass.
  • Lindsey, D. (1980). Production and citation measures in the sociology of science: The problem of multiple authorship. Social Studies of Science, 10, 145-162.[Abstract]
  • Lindsey, D. (1982). Further evidence for adjusting for multiple authorship. Scientometrics, 4, 389-395.[CrossRef]
  • Lindsey, D. (1989). Using citation counts as a measure of quality in science: Measuring what's measurable rather than what's valid. Scientometrics, 15, 189-203.
  • Lindsey, D., & Kirk, S. A. (1992). The role of social work journals in the development of a knowledge base for the profession. Social Service Review, 66, 295-310.
  • MacRoberts, M. H., & MacRoberts, B. R. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40, 342-349.[CrossRef]
  • MacRoberts, M. H., & MacRoberts, B. R. (1997). Citation content analysis of a botany journal. Journal of the American Society for Information Science, 48, 274-275.[CrossRef]
  • Meittunen, J., & Nieminen, P. (2003). The effect of statistical methods and study reporting characteristics on the number of citations: A study of four general psychiatric journals. Scientometrics, 57, 377-388.[CrossRef]
  • Moed, H. F. (2005). Citation analysis in research evaluation. Dordrecht, Netherlands: Springer.
  • Monastersky, R. (2005a). Impact factors run into competition. Chronicle of Higher Education, 52(8), A17-A17.
  • Monastersky, R. (2005b). The number that's devouring science. Chronicle of Higher Education, 52(8), A12-A12.
  • Oromaner, M. (1974). Collaboration and impact: The career of multiauthored publications. Social Science Information, 14, 147-155.
  • Ouwehand, A. (2002). Power calculator. Retrieved March 31, 2005, from http://calculators.stat.ucla.edu/powercalc/
  • Pardeck, J. T. (1992). Are social work journal editorial boards competent? Some disquieting data with implications for research on social work practice. Research on Social Work Practice, 2, 487-496.[Abstract/Free Full Text]
  • Pardeck, J. T., Arndt, B. J., Light, D. B., Mosley, G. F., Thomas, S. D., Werner, M. A., et al. (1991). Distinction and achievement levels of editorial board members of psychology and social work journals. Psychological Reports, 68, 523-527.
  • Pardeck, J. T., Chung, W. S., & Murphy, J. W. (1995). An examination of the scholarly productivity of social work journal editorial board members and guest reviewers. Research on Social Work Practice, 5, 223-234.[Abstract/Free Full Text]
  • Pardeck, J. T., & Meinart, R. G. (1999). Scholarly achievements of the Social Work editorial board and consulting editors: A commentary. Research on Social Work Practice, 9, 86-91.[Abstract]
  • Phelan, T. J. (1999). A compendium of issues for citation analysis. Scientometrics, 45, 117-136.[Medline] [Order article via Infotrieve]
  • Plomp, R. (1990). The significance of the number of highly cited papers as an indicator of scientific prolificacy. Scientometrics, 19, 185-197.[CrossRef]
  • Porter, A. L. (1977). Citation analysis: Queries and caveats. Social Studies of Science, 7, 257-267.[Free Full Text]
  • Rhoades, G. (1998). Managed professionals: Unionized faculty and restructuring academic labor. Albany: State University of New York Press.
  • Rhoads, R. A., & Rhoades, G. (2005). Graduate employee unionization as symbol of and challenge to the corporatization of U.S. research universities. Journal of Higher Education, 76 (3), 243-275.[CrossRef]
  • Rosenberg, G., Holden, G., & Barker, K. (2005). What happens to our ideas? A bibliometric analysis of articles in Social Work in Health Care in the 1990s. Social Work in Health Care, 41, 35-66.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91, 42-46.[Medline] [Order article via Infotrieve]
  • SAS Institute Inc. (2003). SAS 9.1 for Windows [Computer program]. Cary, NC: SAS Institute.
  • Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.[ISI]
  • Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43 (9), 628-638.[CrossRef]
  • Seglen, P. O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45 (1), 1-11.[Medline] [Order article via Infotrieve]
  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498-502. Retrieved February 17, 2003, from http://bmj.com/cgi/content/full/314/7079/497[Free Full Text]
  • Seipel, M. M. O. (2003). Assessing publication for tenure. Journal of Social Work Education, 39, 79-88.
  • Sellers, S. L., Mathiesen, S. G., Perry, R., & Smith, T. (2004). Evaluation of social work journal quality: Citation versus reputation approaches. Journal of Social Work Education, 40, 143-160.
  • Sevinc, A. (2004). Manipulating impact factor: An unethical issue or an editor's choice. Swiss Medical Weekly, 134, 410-410.[Medline] [Order article via Infotrieve]
  • Slaughter, S., & Leslie, L. L. (1997). Academic capitalism: Politics, policies, and the entrepreneurial university. Baltimore: Johns Hopkins University Press.
  • Smith, R. (1997). Journal accused of manipulating impact factor. British Medical Journal, 314, 461-461.[Free Full Text]
  • Smith, R. (1998). Unscientific practice flourishes in science. British Medical Journal, 316, 1036-1040.[Free Full Text]
  • Snijders, T. A. B., & Bosker, R. J. (1994). Modeled variance in two-level models. Sociological Methods & Research, 22, 342-363.[Abstract]
  • Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
  • Staff. (1999). Citations reveal concentrated influence: Some fields have it, but what does it mean? Science Watch, 10(1), 1-2.
  • Staff. (2004b). The undue importance of the impact factor. European Journal of Cancer, 40, 919-919.
  • Statistical Package for the Social Sciences. (2004). SPSS for Windows, 13.0. Chicago: Author.
  • Tenure Modifications. (2005, September). Copy editor. New York University: Faculty Senators Council, p. 3-3.
  • Thyer, B. A. (2004). Evaluating social work journal publication systems. Journal of Social Work Education, 40, 361-363.
  • Urbaniak, G. C., & Plous, S. (2005). Research randomizer. Retrieved July 31, 2005, from http://www.randomizer.org/
  • Weale, A. R., Bailey, M., & Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: A comparison to the impact factor. BMC Medical Research Methodology, p. 4-4. Retrieved July 15, 2005, from http://www.biomedcentral.com/1471-2288/4/14
  • Weigel, R. J., Dracon, G., Radhakrishnan, R., Rho, Y., Sevgen, F., & Dafoe, D. C. (2004). Incentive systems for academic productivity in a depart of surgery. Journal of the American College of Surgeons, 199, 300-307.[CrossRef][ISI][Medline] [Order article via Infotrieve]
  • Whitehouse, G. H. (2001). Citation rates and impact factors: Should they matter? British Journal of Radiology, 74, 1-3.[Free Full Text]
  • Zwahlen, M., Junker, C., & Egger, M. (2004). Commentary II. The journal impact factor in the evaluation of research quality: Villain, scapegoat or innocent bystander? Sozial Und Praventivmedizin, 49, 1, 19-22.[Medline] [Order article via Infotrieve]

Add to CiteULike CiteULike   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to Technorati Technorati    What's this?


This article has been cited by other articles:


Home page
 Research on Social Work PracticeHome page
G. Holden, K. Barker, L. Covert-Vail, G. Rosenberg, and S. A. Cohen
Does Social Work Abstracts Work?
Research on Social Work Practice, September 1, 2008; 18(5): 487 - 499.
[Abstract] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Free Full Text (Free PDF) Free
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in ISI Web of Science
Right arrow Alert me to new issues of the journal
Right arrow Add to Saved Citations
Right arrow Download to citation manager
Right arrowRequest Permissions
Right arrow Request Reprints
Right arrow Add to My Marked Citations
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Holden, G.
Right arrow Articles by Onghena, P.
Right arrow Search for Related Content
Social Bookmarking
Add to CiteULike   Add to Connotea   Add to Del.icio.us   Add to Digg   Add to Reddit   Add to Technorati  
What's this?