際際滷

際際滷Share a Scribd company logo

Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance

Download as PPTX, PDF
Lorna Wildgaard
Lorna Wildgaard

PhD defense about the appropriate use and validity of bibliometric indicators at the individual level. The presentation considers the familial relationships between indicators, the extent concepts being measured are captured by the indicator model as well as the general characteristics of the indicators and the profile of the researchers they are designed to measure

1 of 32
Downloaded 12 times
Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance Lorna Wildgaard Royal School of Library and Information Science Faculty of the Humanities, CPH U 27th November 2015
OVERVIEW 1. Background 2. Characteristics and effects of ALI (Papers 1, 2, 4) 3. The appropriateness of ALI across different disciplines and different academic seniorities (Papers 3, 5, 6, 7) 4. Concept definintion in the construction of ALI (Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca) 5. Conclusions Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CORRECTION
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion Bibliometrics is the application of various statistical analyses to study patterns of authorship, publication, and literature use. (Lancaster, 1977). [] to analyse the structure of science, measure science, and indicate the production, citations and collaboration of researchers, institutions and countries. (De Bellis 2014; Pritchard 1969).
4 1980s 1990s 2000s 2010s1970s1960s1950s de Solla Price growth model Namilov interdisciplinary approach to indexing scientific literature Merton Social structure of scientific communication Haitun; Yablonsky distribution of citations Garfield SCI & IF h-index Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
5 Managerial Teaching Outreach Societal Technical Skills Funding/Grants Productivity Quality Relevance Visibility Reputation Scientific Counting, adding, dividing, multiplying an authors publications & citations Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
DEFINTITIONS OF AU, P & C IN ALI Authorship, publication and citation is different from discipline to discipline, from time to time and location to location and has implications in indicator development (Bo邸njak and Maru邸i, 2012) What constitutes a publication [author and citation] should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012) Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
EVALUATION OUTPUT DATA ALI MODEL MOTIVATION TO PUBLISH MOTIVATION TO CITE INTERPRETATION OF ALI LEGITIMATE LINK TO REAL WORLD PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS EXOGENOUS VARIABLES WHAT IS MISSING INDEXING POLICY Background Preliminary Analyses Empirical Analyses Validation Study Conclusion THE PROBLEM
1. What are the characterisitics of ALI of academic performance? 2. To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? 3. To what extent are the concepts being measured defined in the construction of ALI? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RESEARCH QUESTIONS
9 ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF- WORTH ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS DOCUMENTING BEING OUT- PERFORMED IS DETRIMENTAL TO A RESEARCHERS SELF-DEFINITION ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION SUCCESS IS DEFINED AS WELL IN THE SYSTEM EXTERNAL CHARACTERISTICS OF ALI
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion INTERNAL CHARACTERISTICS OF ALI
ALI are designed to measure particular aspects of the effect of a researchers work over time, to field, as quality, ranking all or selected works, co- authorship etc. Judgements based on ALI can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual. ALI have to be methodologically sound PRELIMINARY ANALYSIS: RESULTS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
FAMILY CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion METHOD: Compute simple indicators on publication and citation data of 750 researchers (585 m, 165w) 4 disciplines, 5 academic seniorities Correlation (Kendalls Tau) as measure of association between seniority and ALI, discipline by discipline ALSCAL IBM SPSS v.19 ENDPOINT Suitability of indicators in different disciplines and seniorities
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion FAMILY CHARACTERISTICS DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCE ASTRONOMY hg 0.37 25 ENVIRONMENTAL SCIENCE h, h2 0.37 24 PHILOSOPHY IQP 0.38 47 PUBLIC HEALTH g 0.49 38
METHOD: Publication and citation data for 512 researchers 22,143 papers, 423, 371 citations (WOS) 52,227 papers, 746,985 citations (GS) For each researcher 17 ALI calculated Agreement: determined by matching rank positions Variability of rank position determined by standard deviation of the difference in scholar rank position, calculated from the matched pairs END POINTS: agreement in ranking between WOS & GS RANKING CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
Background Preliminary Analyses Empirical Analyses Validation Study Conclusion hg produced highest level of agreement in rankings in WOS & GS, 0.8-0.9 tau across disciplines and seniorities (Patel 2013)
hg fulfills its potential as a ranking indicator (Alonso et al 2010) yet this tells us nothing about the excellence of the researcher. h and g cannot be rationally combined to produce indications of research excellence (Franceschini & Maisano 2011). Indicator rankings inform of the researchers visibility in the citation index not in their academic community (Bar-Ilan 2008; De Battisti 2012; Farhadi et al 2013). Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RANKING CHARACTERISTICS
METHOD: Same data as previous studies Explorative statistical study of distribution; two step cluster; ordinal regression: odds ratios IBM SPSS v.22 ENDPOINT: Identify disciplinary & seniority appropriate indicators Background Preliminary Analyses Empirical Analyses Validation Study Conclusion GROUPING CHARACTERISTICS
CLUSTER 1 below IQR, high nnc, sc, low P & C. Adjusting for academic age mg, m.quotient, currency of cites improved scores. CLUSTER 2 Median IQR, high scores on collaboration indicators CLUSTER 3 Top IQR, highest scores normalized for field, one SIG paper CLUSTER 4 Extreme outliers, highest C, cited quickly, highest htype scores
ASTRO HEALTHPHILENVIRO h2 SUM PP TOP PROP Q2 e GROUPING CHARACTERISTICS Ordinal regression: Academic age is statistically significant for cluster placement Within cluster rank position determined by the ratio P:h
1. Simple ALI are complex! 2. Basic descriptive statistics can be more informative 3. There are families of indicators 4. ALI scores are highly influenced by the database, academic age and the fit of the data to the model 5. ALI are more appropriate in some disciplines than others 6. Different ALI produce stable rankings within and across databases. 7. ALI can be easily manipulated 8. Scores can be due to chance rather than practical importance. EMPIRICAL ANALYSIS: CONCLUSIONS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
Knowing all these practical challenges, what properties should a well constructed indicator contain? METHOD: Based on papers 1-7 Validate ALI using Gingras criteria Analyze operationalization of AU, P, C in ALI ENDPOINT: Recommendation of a set of disciplinary dependent indicators of scientific publication output Background Preliminary Analyses Empirical Analyses Validation Study Conclusion VALIDITY ANALYSIS
THE 3 CRITERIA OF PROFESSOR YVES GINGRAS 1. Adequacy of the indicator to the object it measures -quantity of items a researcher publishes is an indicator of production 2. Sensitivity to the intrinsic inertia of the object -changing values in the time interval correspond to the speed and direction of change in the object 3. Homogeneity of the dimensions of the indicator -h-index is a heterogeneous indicator: it mixes quantity of papers published with citation counts Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
10 28 HOMOGENEITY ADEQUACY INERTIA Background Preliminary Analyses Empirical Analyses Validation Study Conclusion 17 did not fulfill any of the criteria n=69 ALI 4 1 7 20
OPERATIONALIZATION OF AU, P, C THE LOGIC GRID METHOD: Extract definitions and hypotheses in ALI Matrix of definitions of AU, P, C & aim of ALI AIM: Identify in a simple way the properties of ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER CITATION 6 6 6 5 2 PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF. 2 1 1 1 1 20 PUBLISHED SCIENTIST AUTHOR AWARD WINNER PUBLISHED & CITED SENIORITY NO DEF. AUTHOR 12 11 10 7 5 2 5 PAPER IN WOS PAPER IN OTHER INDEX PAPER OBJECTS WITH CITATION EXPRESSION PUBLICATIONS 30 15 6 1 1 EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY MEASURE 9 7 7 6 6 4 GROWTH QUALITY DISTRIBUTION QUANTITY & QUALITY CAREER DURABILITY 3 3 2 2 1 1
LOGIC GRID http://tinyurl.com/ofm7h8s
ALI are designed for specific author and publication types Majority of ALI designed for papers in WoS Some indicators are purely theoretical Inherent bias in all ALI is that they are designed for researcher profiles befitting the hard sciences Lack of definitions = methodological deficiency If developers cannot define the variables they are measuring, how can we? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
1. Transparency Know which data is used to compute the indicator Understand the math and the inferences 2. Demographics Be aware if demographics affect the ALI scores 3. Motive ALI must fit the objectives of the evaluation 4. Diversity Choose ALI that fit the discipline 5. Openess Make the limitations of ALI explicit, use supplementary ALI PROPERTIES OF A WELL CONSTRUCTED ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
RQ 1: What are the characterisitics of ALI of academic performance? ALI are complex in character. They use different: disciplinary perspectives, objectives, operationalization of variables, requirements to data, and mathematical models that favour different disciplines and seniorities. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS one size does NOT fit all.
RQ 2: To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? No seniority ALI were identified There is a disconnection between the performance of researcher on the CV and in ALI. Some ALI are more appropriate in some disciplines than others as grouping and ranking indicators or as indicators that isolate unique information Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
RQ 3: To what extent are the concepts being measured defined in the construction of ALI? Many indicators are poorly or only partially defined Decisions based on these ALI could turn out to be less effective than expected If we do not know how the developer has done the measurement, we cannot repeat it, reflect on the meaning associated with the concept and compare results to previous findings Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
THANK YOU!! ACUMEN partners: especially Judit Bar-Ilan, Frank van der Most & Paul Wouters. My supervisors: Jesper Schneider, Peter Ingwersen & Birger Larsen PhD/Masters group: Jesper, Niels-Peder, Maria, Ole, Rikke, Sara, Sille, Lisa Sutherland & BADASS Researcher/teacher support: Johanne Maibohm, Susanne Acevedo, Pia Dithmar & Ragnhild Riis. My boys: Kim, Zander & Balthazar My opponents: Henk Moed Rodrigo Costas & Jeppe Nicolaisen

More Related Content

Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance

  • 1. Measure up! The extent author-level bibliometric indicators are appropriate measures of individual researcher performance Lorna Wildgaard Royal School of Library and Information Science Faculty of the Humanities, CPH U 27th November 2015
  • 2. OVERVIEW 1. Background 2. Characteristics and effects of ALI (Papers 1, 2, 4) 3. The appropriateness of ALI across different disciplines and different academic seniorities (Papers 3, 5, 6, 7) 4. Concept definintion in the construction of ALI (Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca) 5. Conclusions Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CORRECTION
  • 3. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion Bibliometrics is the application of various statistical analyses to study patterns of authorship, publication, and literature use. (Lancaster, 1977). [] to analyse the structure of science, measure science, and indicate the production, citations and collaboration of researchers, institutions and countries. (De Bellis 2014; Pritchard 1969).
  • 4. 4 1980s 1990s 2000s 2010s1970s1960s1950s de Solla Price growth model Namilov interdisciplinary approach to indexing scientific literature Merton Social structure of scientific communication Haitun; Yablonsky distribution of citations Garfield SCI & IF h-index Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 6. DEFINTITIONS OF AU, P & C IN ALI Authorship, publication and citation is different from discipline to discipline, from time to time and location to location and has implications in indicator development (Bo邸njak and Maru邸i, 2012) What constitutes a publication [author and citation] should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012) Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 7. EVALUATION OUTPUT DATA ALI MODEL MOTIVATION TO PUBLISH MOTIVATION TO CITE INTERPRETATION OF ALI LEGITIMATE LINK TO REAL WORLD PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS EXOGENOUS VARIABLES WHAT IS MISSING INDEXING POLICY Background Preliminary Analyses Empirical Analyses Validation Study Conclusion THE PROBLEM
  • 8. 1. What are the characterisitics of ALI of academic performance? 2. To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? 3. To what extent are the concepts being measured defined in the construction of ALI? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RESEARCH QUESTIONS
  • 9. 9 ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF- WORTH ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS DOCUMENTING BEING OUT- PERFORMED IS DETRIMENTAL TO A RESEARCHERS SELF-DEFINITION ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION SUCCESS IS DEFINED AS WELL IN THE SYSTEM EXTERNAL CHARACTERISTICS OF ALI
  • 10. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion INTERNAL CHARACTERISTICS OF ALI
  • 11. ALI are designed to measure particular aspects of the effect of a researchers work over time, to field, as quality, ranking all or selected works, co- authorship etc. Judgements based on ALI can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual. ALI have to be methodologically sound PRELIMINARY ANALYSIS: RESULTS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 12. FAMILY CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion METHOD: Compute simple indicators on publication and citation data of 750 researchers (585 m, 165w) 4 disciplines, 5 academic seniorities Correlation (Kendalls Tau) as measure of association between seniority and ALI, discipline by discipline ALSCAL IBM SPSS v.19 ENDPOINT Suitability of indicators in different disciplines and seniorities
  • 13. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion FAMILY CHARACTERISTICS DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCE ASTRONOMY hg 0.37 25 ENVIRONMENTAL SCIENCE h, h2 0.37 24 PHILOSOPHY IQP 0.38 47 PUBLIC HEALTH g 0.49 38
  • 14. METHOD: Publication and citation data for 512 researchers 22,143 papers, 423, 371 citations (WOS) 52,227 papers, 746,985 citations (GS) For each researcher 17 ALI calculated Agreement: determined by matching rank positions Variability of rank position determined by standard deviation of the difference in scholar rank position, calculated from the matched pairs END POINTS: agreement in ranking between WOS & GS RANKING CHARACTERISTICS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 15. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion hg produced highest level of agreement in rankings in WOS & GS, 0.8-0.9 tau across disciplines and seniorities (Patel 2013)
  • 16. hg fulfills its potential as a ranking indicator (Alonso et al 2010) yet this tells us nothing about the excellence of the researcher. h and g cannot be rationally combined to produce indications of research excellence (Franceschini & Maisano 2011). Indicator rankings inform of the researchers visibility in the citation index not in their academic community (Bar-Ilan 2008; De Battisti 2012; Farhadi et al 2013). Background Preliminary Analyses Empirical Analyses Validation Study Conclusion RANKING CHARACTERISTICS
  • 17. METHOD: Same data as previous studies Explorative statistical study of distribution; two step cluster; ordinal regression: odds ratios IBM SPSS v.22 ENDPOINT: Identify disciplinary & seniority appropriate indicators Background Preliminary Analyses Empirical Analyses Validation Study Conclusion GROUPING CHARACTERISTICS
  • 18. CLUSTER 1 below IQR, high nnc, sc, low P & C. Adjusting for academic age mg, m.quotient, currency of cites improved scores. CLUSTER 2 Median IQR, high scores on collaboration indicators CLUSTER 3 Top IQR, highest scores normalized for field, one SIG paper CLUSTER 4 Extreme outliers, highest C, cited quickly, highest htype scores
  • 19. ASTRO HEALTHPHILENVIRO h2 SUM PP TOP PROP Q2 e GROUPING CHARACTERISTICS Ordinal regression: Academic age is statistically significant for cluster placement Within cluster rank position determined by the ratio P:h
  • 20. 1. Simple ALI are complex! 2. Basic descriptive statistics can be more informative 3. There are families of indicators 4. ALI scores are highly influenced by the database, academic age and the fit of the data to the model 5. ALI are more appropriate in some disciplines than others 6. Different ALI produce stable rankings within and across databases. 7. ALI can be easily manipulated 8. Scores can be due to chance rather than practical importance. EMPIRICAL ANALYSIS: CONCLUSIONS Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 21. Knowing all these practical challenges, what properties should a well constructed indicator contain? METHOD: Based on papers 1-7 Validate ALI using Gingras criteria Analyze operationalization of AU, P, C in ALI ENDPOINT: Recommendation of a set of disciplinary dependent indicators of scientific publication output Background Preliminary Analyses Empirical Analyses Validation Study Conclusion VALIDITY ANALYSIS
  • 22. THE 3 CRITERIA OF PROFESSOR YVES GINGRAS 1. Adequacy of the indicator to the object it measures -quantity of items a researcher publishes is an indicator of production 2. Sensitivity to the intrinsic inertia of the object -changing values in the time interval correspond to the speed and direction of change in the object 3. Homogeneity of the dimensions of the indicator -h-index is a heterogeneous indicator: it mixes quantity of papers published with citation counts Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 23. 10 28 HOMOGENEITY ADEQUACY INERTIA Background Preliminary Analyses Empirical Analyses Validation Study Conclusion 17 did not fulfill any of the criteria n=69 ALI 4 1 7 20
  • 24. OPERATIONALIZATION OF AU, P, C THE LOGIC GRID METHOD: Extract definitions and hypotheses in ALI Matrix of definitions of AU, P, C & aim of ALI AIM: Identify in a simple way the properties of ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 25. HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER CITATION 6 6 6 5 2 PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF. 2 1 1 1 1 20 PUBLISHED SCIENTIST AUTHOR AWARD WINNER PUBLISHED & CITED SENIORITY NO DEF. AUTHOR 12 11 10 7 5 2 5 PAPER IN WOS PAPER IN OTHER INDEX PAPER OBJECTS WITH CITATION EXPRESSION PUBLICATIONS 30 15 6 1 1 EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY MEASURE 9 7 7 6 6 4 GROWTH QUALITY DISTRIBUTION QUANTITY & QUALITY CAREER DURABILITY 3 3 2 2 1 1
  • 27. ALI are designed for specific author and publication types Majority of ALI designed for papers in WoS Some indicators are purely theoretical Inherent bias in all ALI is that they are designed for researcher profiles befitting the hard sciences Lack of definitions = methodological deficiency If developers cannot define the variables they are measuring, how can we? Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 28. 1. Transparency Know which data is used to compute the indicator Understand the math and the inferences 2. Demographics Be aware if demographics affect the ALI scores 3. Motive ALI must fit the objectives of the evaluation 4. Diversity Choose ALI that fit the discipline 5. Openess Make the limitations of ALI explicit, use supplementary ALI PROPERTIES OF A WELL CONSTRUCTED ALI Background Preliminary Analyses Empirical Analyses Validation Study Conclusion
  • 29. RQ 1: What are the characterisitics of ALI of academic performance? ALI are complex in character. They use different: disciplinary perspectives, objectives, operationalization of variables, requirements to data, and mathematical models that favour different disciplines and seniorities. Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS one size does NOT fit all.
  • 30. RQ 2: To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities? No seniority ALI were identified There is a disconnection between the performance of researcher on the CV and in ALI. Some ALI are more appropriate in some disciplines than others as grouping and ranking indicators or as indicators that isolate unique information Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
  • 31. RQ 3: To what extent are the concepts being measured defined in the construction of ALI? Many indicators are poorly or only partially defined Decisions based on these ALI could turn out to be less effective than expected If we do not know how the developer has done the measurement, we cannot repeat it, reflect on the meaning associated with the concept and compare results to previous findings Background Preliminary Analyses Empirical Analyses Validation Study Conclusion CONCLUSIONS
  • 32. THANK YOU!! ACUMEN partners: especially Judit Bar-Ilan, Frank van der Most & Paul Wouters. My supervisors: Jesper Schneider, Peter Ingwersen & Birger Larsen PhD/Masters group: Jesper, Niels-Peder, Maria, Ole, Rikke, Sara, Sille, Lisa Sutherland & BADASS Researcher/teacher support: Johanne Maibohm, Susanne Acevedo, Pia Dithmar & Ragnhild Riis. My boys: Kim, Zander & Balthazar My opponents: Henk Moed Rodrigo Costas & Jeppe Nicolaisen