Modern Psychometrics

John Rust is the founder of The Psychometrics Centre at the University of Cambridge, UK. He is a Senior Member of Darwin College, UK, and an Associate Fellow of the Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK. Michal Kosinski is an associate professor of organizational behavior at Stanford Graduate School of Business, USA. David Stillwell is the academic director of the Psychometrics Centre at the University of Cambridge, UK. He is also a reader in computational social science at the Cambridge Judge Business School, UK.

1. The history and evolution of psychometric testing

Introduction

What is psychometrics?

Psychometrics in the 21st century

History of assessment





Chinese origins

The ability to learn

The nineteenth century





Beginnings of psychometrics as a science





Intelligence testing

Eugenics and the dark decades





Psychometric testing of ability





The dark ages come to an end

An abundance of abilities





Tests of other psychological constructs





Personality

Integrity

Interests

Motivation

Values

Temperament

Attitude

Belief





Summary

2. Constructing your own psychometric questionnaire

The purpose of the questionnaire

Making a blueprint

Writing items





Alternate-choice items

Multiple-choice items

Rating-scale items

All questionnaires

Knowledge-based questionnaires

Person-based questionnaires





Designing the questionnaire

Piloting the questionnaire

Item analysis





Facility

Discrimination

Distractors





Obtaining the reliability





Cronbach’s alpha

Split-half reliability





Assessing validity





Face validity

Content validity





Standardization

3. The Psychometric principles

Reliability





Test-retest reliability

Parallel-forms reliability

Split-half reliability

Interrater reliability

Internal consistency

The standard error of measurement (SEM)

Comparing test reliabilities

Restriction of range





Validity





Face validity

Content validity

Predictive validity

Concurrent validity

Construct validity

Differential validity





Standardization





Norm referencing

Criterion referencing





Equivalence





Differential item functioning

Measurement invariance

Adverse impact





Summary

4. Psychometric measurement

True-score theory

Identification of latent traits with factor analysis





Spearman’s two-factor theory

Vector algebra and factor rotation

Moving into more dimensions

Multidimensional scaling





Application of factor analysis to test construction





Eigenvalues

Identifying the number of factors to extract using the Kaiser criterion

Identifying the number of factors to extract using the Cattell scree test

Other techniques for identifying the number of factors to extract

Factor rotation

Rotation to simple structure

Orthogonal rotation

Oblique rotation





Limitations of the classical factor-analystic approach

Criticisms of psychometric measurement theory





The Platonic true score

Psychological vs. physical true scores

Functional assessment and competency testing

Machine learning and the black box





Summary

5. Item response theory and computer adaptive testing

Introduction

Item banks





The Rasch model

Assessment of educational standards

The Birnbaum model





The evolution of modern psychometrics





Computer adaptive testing

Item equating

Polytomous IRT





An intuitive graphical description of item tesponse theory





Limitations of classical test theory





A graphical Introduction to item response theory





The logistic curve

3PL-model: difficulty parameter

3PL model: discrimination parameter

3PL model: guessing parameter

The Fisher information function

The test information function and its relationship to the standard error of measurement

How to score an IRT test





Principles of computer adaptive testing

Summary of item response theory

Confirmatory factor analysis

6. Personality theory

Theories of personality





Psychoanalytic theory

Humanistic theory

Social learning theory

Behavioral genetics

Type and trait theories

Different approaches to personality assessment









Self-report techniques and personality profiles

Reports by others

Online digital footprints

Situational assessments

Projective measures

Observations of behavior

Task performance methods

Polygraph methods

Repertory grids





Sources and management of bias





Self-report techniques and personality profiles

Reports by others

Online digital footprints

Situational assessments

Projective measures

Observations of behavior

Task performance methods

Polygraph methods

Repertory grids





Informal methods of personality assessment

State versus trait measures

Ipsative scaling

Spurious validity and the Barnum Effect

Summary

7. Personality assessment in the workplace

Prediction of successful employment outcomes





Validation of personality questionnaires previously used in employment

Historical antecedents to the five-factor model

Stability of the five-factor model

Cross-cultural aspects of the five-factor model

Scale independence and the role of facets

Challenges to scale construction for the five-factor model

Impression management

Acquiescence

Response bias and factor structure

Development of the five OBPI personality scales





Assessing counterproductive behavior at work





The impact of behaviorism

Prepsychological theories of integrity

Modern integrity testing

Psychiatry and the medical model

The dysfunctional tendencies

The dark triad

Assessing integrity at work

The OBPI integrity scales





Conclusion

8. Employing digital footprints in psychometrics

Introduction

Types of digital footprint





Usage logs

Language data

Mobile sensors

Images and audiovisual data





Typical applications of digital footprints in psychometrics





Replacing and complimenting traditional measures

New contexts and new constructs

Predicting future behavior

Studying human behavior

Supporting the development of traditional measures





Advantages and challenges of employing digital footprints in psychometrics





High ecological validity

Greater detail and longitude

Less control over the assessment environment

Greater speed and unobtrusiveness

Less privacy and control

No anonymity

Bias

Enrichment of existing constructs

Developing digital-footprint-based psychometric measures
Collecting digital footprints









How much data is needed?





Preparing digital footprints for analysis





Respondent-footprint matrix

Data sparsity





Reducing the dimensionality of the respondent-footprint matrix





Singular value decomposition

Latent Dirichlet allocation





Building prediction models

9. Psychometrics in the era of the intelligent machine

History of computerization in psychometrics





Computerized statistics

Computerized item banks

Computerized item generation

Automated advice and report systems





The evolution of AI in psychometrics





Expert systems

Neural networks (machine learning)

Parallel processing

Predicting with statistics and machine learning

Explainability





Psychometrics in cyberspace





What and where is cyberspace?

The medium is the message





Moral development in AI





Kohlberg’s theory of moral development

Do machines have morals?

The laws of robotics

Artificial general intelligence





Conclusion