Citation Analysis in Research Evaluation (eBook)

Contents 6
Preface 9
EXECUTIVE SUMMARY 14
PART 1 GENERAL INTRODUCTION AND MAIN CONCLUSIONS 22
1 GENERAL INTRODUCTION 24
1.1 The ISI Citation Indexes as search and research tools 24
1.2 Quantitative science and technology studies 28
1.3 Scope and structure of the book 32
2 BASIC NOTIONS AND GENERAL CONCLUSIONS 38
2.1 Introduction 38
2.2 About the nature of the concept of scholarly research quality 38
2.3 Characteristics of research evaluation and policy processes 39
2.4 Use of bibliometric indicators in research evaluation and policy 41
2.5 The role of bibliometric investigators 43
2.6 The role of scholars subjected to citation analysis 44
2.7 The role of evaluators 45
2.8 Citation analysis and peer review 46
3 SYNOPSIS 48
Part 2.1 Assessing basic science research departments and scientific journals 48
Part 2.2 The ISI Citation Indexes 54
Part 2.3 Assessing research performance in social sciences and humanities 57
Part 2.4 Accuracy issues 60
Part 2.5 Theoretical aspects 63
Part 2.6 Citation analysis and peer review 68
Part 2.7 Macro studies 72
Part 2.8 New developments 78
Part 2 Empirical and Theoretical Chapters 83
PART 2.1 ASSESSING BASIC SCIENCE RESEARCH DEPARTMENTS AND SCIENTIFIC JOURNALS 83
CITATION ANALYSIS OF BASIC SCIENCE RESEARCH DEPARTMENTS 84
4.1 Introduction 84
4.2 Data collection and accuracy 89
4.3 ISI Citation Indexes: coverage, biases 91
4.4 General validity issues 93
4.5 Indicators and their validity 97
4.6 General issues of interpretation and use 100
CITATION ANALYSIS OF SCIENTIFIC JOURNALS 103
5.1 Introduction 103
5.2 Issues regarding ISI/JCR impact factor 104
5.3 Normalised journal impact measures 112
5.4 Further issues and conclusions 114
PART 2.2 THE ISI CITATION INDEXES 120
BASIC PRINCIPLES, CITATION LINKS AND TERMINOLOGY 122
6.1 Introduction 122
6.2 Basic principles 123
6.3 Citation links and terminology 127
ISI COVERAGE BY DISCIPLINE 132
7.1 Introduction 132
7.2 Some earlier studies of ISI coverage 133
7.3 Coverage indicators and their interpretation 135
7.4 Results at an aggregate level 136
7.5 Results per discipline 138
7.6 Journal internationality by discipline 144
7.7 Discussion and conclusions 145
IMPLICATIONS FOR THE USE OF THE ISI CITATION INDEXES IN RESEARCH EVALUATION 150
8.1 Introduction 150
8.2 Four types of research assessment studies 152
8.3 Examples 155
PART 2.3 ASSESSING SOCIAL SCIENCES AND HUMANITIES 158
DIFFERENCES BETWEEN SCIENCE, SOCIAL SCIENCES AND HUMANITIES 160
9.1 Introduction 160
9.2 Methodology for the development of research performance indicators in social sciences and humanities 164
EXPANDED CITATION ANALYSIS: A CASE STUDY IN ECONOMICS 166
10.1 Introduction 166
10.2 Citation impact indicators for documents in non-ISI covered media 167
10.3 Results 168
10.4 Concluding remarks 170
A CASE STUDY OF RESEARCH PERFORMANCE IN LAW 172
11.1 Introduction 172
11.2 Analysis of earlier assessments 173
11.3 Publication lists and classifications 174
11.4 Analysis of classifications and quality perceptions 175
11.5 Comments by Deans of law departments 178
PART 2.4 ACCURACY ASPECTS 181
INTRODUCTORY NOTES ON ACCURACY ISSUES 182
12.1 Introduction 182
12.2 Notes by Eugene Garfield 183
ACCURACY OF CITATION COUNTS 186
13.1 Introduction and research questions 186
13.2 Data and methods 187
13.3 Results 188
13.4 Discussion and conclusions 192
PROBLEMS WITH THE NAMES OF AUTHORS AND INSTITUTIONS, AND WITH THE DELIMITATION OF SUBFIELDS 194
14.1 Author names 194
14.2 Institutional affiliations 196
14.3 Discipline and subfield delimitations 200
PART 2.5 THEORETICAL ASPECTS 204
WHAT DO REFERENCES AND CITATIONS MEASURE? 206
15.1 Introduction 206
15.2 Disciplinary approaches 208
15.3 Views of scholars 211
TOWARDS A THEORY OF CITATION: SOME BUILDING BLOCKS 222
16.1 Introduction 222
16.2 Reference lists are selective and contain both unique and more commonly used cited references 224
16.3 Extreme positions are not useful in the debate on citation theories 226
16.4 Comments on the views of scholars discussed in Chapter 15 227
16.5 Research articles are elements of publication ensembles of research groups carrying out aresearch programme 229
16.6 A reference list constitutes a distinct part of a paper with proper functions 231
IMPLICATIONS FOR THE USE OF CITATION ANALYSIS IN RESEARCH EVALUATION 234
17.1 What do citations measure? 234
17.2 Implications 237
PART 2.6 CITATION ANALYSIS AND PEER REVIEW 240
PEER REVIEW AND THE USE AND VALIDITY OF CITATION ANALYSIS 242
18.1 Introduction 242
18.2 About peer review 242
18.3 Use of bibliometric indicators within peer review 246
18.4 Validation of bibliometric indicators regarding individual scholars and research departments 247
ANALYSIS OF PEER ASSESSMENTS OF RESEARCH DEPARTMENTS 252
19.1 Introduction 252
19.2 Data and methods 253
19.3 Results 254
19.4 Discussion and conclusions 257
Notes 258
ANALYSIS OF A NATIONAL RESEARCH COUNCIL 260
20.1 Introduction 260
20.2 Data and methods 261
20.3 Results 263
20.4 Discussion and conclusions 270
PART 2.7 MACRO STUDIES 272
DID GLOBAL SCIENTIFIC PUBLICATION PRODUCTIVITY INCREASE DURING THE 1980s AND 1990s? 274
21.1 Introduction 274
21.2 Further analysis at an aggregate level 276
21.3 Analysis by discipline 278
21.4 Discussion and conclusions 280
Notes 282
MEASURING TRENDS IN NATIONAL PUBLICATION OUTPUT 283
22.1 Introduction 283
22.2 Difficulties in constructing and interpreting publication based macro indicators 285
22.3 Eleven indicators and their interpretation 287
22.4 Conclusions 290
22.5 Afterword: Detailed discussion of Table 22.1 291
DOES INTERNATIONAL SCIENTIFIC COLLABORATION PAY? 297
23.1 Introduction 297
23.2 Data, methods and results 298
23.3 Conclusions 302
DO US SCIENTISTS OVERCITE PAPERS FROM THEIR OWN COUNTRY? 303
24.1 Introduction 303
24.2 Data 304
24.3 The factor size of domestic publication output 305
24.4 Other factors 309
24.5 Discussion and conclusions 312
PART 2.8 NEW DEVELOPMENTS 314
DEVELOPMENT OF NEW INDICATORS 316
25.1 Introduction 316
25.2 New indicators 317
ELECTRONIC PUBLISHING, NEW DATABASES AND SEARCH ENGINES 326
26.1 Electronic publishing 326
26.2 New databases and search engines 329
FURTHER RESEARCH 331
27.1 Introduction 331
27.2 Assessment of the actual use of citation analysis and its effects 331
27.3 Rankings versus relational indicators 333
References 336
Index of Keywords, Cited Works and Cited Authors 350