Missing the Mark
Canbury Press (Verlag)
978-1-912454-99-0 (ISBN)
'An important contribution to our thinking.’ – Sixth Form Colleges Association
'An uncomfortable but important read.’ – Headmasters’ and Headmistresses’ Conference
'Everyone in UK education should reflect upon the problems identified in this powerful book' – Higher Education Policy Institute
Every summer one million GCSE and A-Level candidates receive results that define their school years and set them up for their life. But those results are gravely unreliable.
In fact, about one grade in four in England is WRONG. That is 1.5 million grades every year.
An A-Level grade B might have been an A, or even a C, had a different examiner marked the script. Similarly, a GCSE grade 7 might have received a grade 8 or a 6.
For a decade, young people and their friends and families have been unable to grasp the full extent of this randomness. Now, in this definitive and easy to follow book, Dennis Sherwood explains why so many pupils receive final grades that don’t do them justice. And he suggests ways to regain trust, which apply to essay-based exams throughout the world.
Reviews
‘Know an A Level student who you were absolutely sure should nail an A* but ended up with a B? Well, they probably should have got that A* but were a victim of this scandal. Sherwood’s work changed my outlook. Let him change yours too.’ – Robert Campbell, former Chief Executive, Morris Education Trust
‘Dennis has been challenging our thinking about assessment and the awarding of grades for many years, combining detailed research with an engaging manner and clear explanations... this is an important contribution to our thinking.’ – Bill Watkin, Chief Executive, Sixth Form Colleges Association
‘Dennis Sherwood asks the questions about exam grades that no one really wants to answer. His analysis suggests that much of what we think we know about school exams is based at best on wishful thinking and at worst on wilful misrepresentation of statistics. But he also has some positive suggestions for improvement. Missing the Mark is an uncomfortable but important read.’ – Melvyn Roffe, Chair, Headmasters’ and Headmistresses’ Conference
‘Missing a grade can result in university or college applications being rejected. Dennis Sherwood asks the $64,000 question: ‘Are grades reliable enough for the purposes they are supposed to serve?’ This book presents an insightful analysis of this important matter, including the rules introduced in 2016 to reduce the number of appeals, the controversial grading processes in 2020 and 2021 when exams were cancelled, why ‘real’ grades are so unreliable, and some solutions too.’ – Huy Duong, parent
‘Everyone in UK education should reflect upon the problems identified in this powerful book – and then decide what to do about them.’ – Nick Hillman, Director, Higher Education Policy Institute
Anyone with an interest in how examinations are assessed, from those in government, regulators, schools, colleges, universities to employers, teachers, parents and students, should read Dennis Sherwood's incisive analysis. His conclusions will have a profound impact on our idea of the accuracy, reliability and fairness of examinations. – Mike Larkin, Emeritus Professor Queen's University of Belfast and Total Equality For Students
‘Dennis provides a clear, step-by-step outline of what is going so terribly wrong and the easy ways to remedy this.’ – Ollie Green, A-level student
About the author
Dennis Sherwood is a management consultant with experience of solving complex problems. He has a Physics Masters from the University of Cambridge, an MPhil in Molecular Biophysics and Biochemistry from Yale University and a PhD in biology from the University of California in San Diego. After being a consulting partner at Deloitte Haskins + Sells, and Coopers & Lybrand, he became an executive director at Goldman Sachs. He now runs his own business, The Silver Bullet Machine Manufacturing Company Limited, specialising in organisational creativity and innovation. He is author of 14 books.
Extract - Foreword
Gold standard! Well, maybe not! For many years England’s GCSE and A-level qualifications have enjoyed an international reputation as world-leading. They have frequently been cited as ‘gold standard’ examinations. In this book Dennis Sherwood applies forensic analysis, in an accessible format, to one aspect of those qualifications – the grades awarded to each student on results day. His expert commentary leaves us in no doubt that the architecture of reliability is nothing more than a fancy façade on a house that’s built on sand.
This is not a book about whether examinations are the best way to assess authentic learning. That’s a different debate, although there’s evidence here that excessive reliance on end-of-course examinations exacerbates the great grading scandal.
This is also not a book about whether the content of our examination-driven school and college curriculum is well-designed, fit for purpose or sufficiently visionary for the future needs of students. That too is a long overdue discussion which should inform public policy, but Dennis retains his focus on one pressing issue. Are the grades awarded to students at the end of the examination process a reliable indicator of their performance and ability? Can those grades be trusted to determine suitability for advanced academic study or access to employment? Do they serve to differentiate authentically between one student and the next?
We are all familiar with the results day photographs that accompany the headlines in August. Enthusiastic celebrations with beaming smiles. Images that are carefully contrived to align with the supporting text as ‘Camelia’ (or whoever) progresses to a top university with her four A* grades or ‘Daniel’ revealed to be a prodigy as he attains twelve grade 9’s in his GCSEs.
Their results may well be impressive and will certainly open doors towards privileged academic opportunities. But what if the student with AAB is actually no better, in any meaningful sense, than the student with BAC? What if these grades lack the precision that they appear to convey? Is there an element of unreliability in how they are awarded – such that two otherwise identical candidates may as well roll a dice alongside completing their examination paper to determine which, say, of two adjacent grades they may ultimately be awarded?
If Dennis is right – and I think he is – then a great grading scandal unfolds before our eyes every summer...
[Buy the book to continue reading the foreword]
Dr Robin Bevan, Headteacher, Southend High School for Boys and NEU Past National President, 2020-21
Dennis Sherwood is a management consultant with experience of solving complex problems. He has a Physics Masters from the University of Cambridge, an MPhil in Molecular Biophysics and Biochemistry from Yale University and a PhD in biology from the University of California in San Diego. After being a consulting partner at Deloitte Haskins + Sells, and Coopers & Lybrand, he became an executive director at Goldman Sachs. He now runs his own business, The Silver Bullet Machine Manufacturing Company Limited, specialising in organisational creativity and innovation. He is author of 14 books. Dr Robin Bevan is Headteacher of Southend High School for Boys. He was National President of the National Education Union in 2020-21.
DEDICATION. To the unknown, but very large, number of young people who have been damaged by the award of wrong exam grades, in the hope that this will not happen in the future
STATISTICS. Over the decade from 2010 to 2019, a total of about 70 million GCSE, AS and A-level grades were awarded following each year’s summer exams in England. • Of which around 17.5 million were wrong. • Yes,17.5 million. • That’s about 1 wrong grade in every 4.
FOREWORD. Foreword by Dr Robin Bevan Headteacher, Southend High School for Boys NEU Past National President, 2020-21. old standard! Well, maybe not! For many years England’s GCSE and A-level qualifications enjoy an international reputation as world-leading. This book forensically analyses grades
1. EXAM GRADES ARE IMPORTANT. A'Level and GCSE grades can affect life chances, yet the regulator Ofqual's own statistics show that 1 in 4 grades can be wrong. This book gives all the evidence, discusses the implications, and – most importantly – offers some solutions
2. EXAMS IN ENGLAND. Deals with GCSE, AS-Level and A-level exams, exam centres and schools, awarding bodies, the regulators Ofsted, DfE, and Ofqual, the House of Commons Education Committee, marking, grade structures, grade boundaries, criterion referencing, cohort referencing, norm referencing
3. ARE EXAM GRADES 99.2% ACCURATE? Edexcel’s claim that grades are 99.2% accurate on results day (taken from Pearson-Edexcel’s website), assesses comments made about grade reliability by School Standards Minister Nick Gibb, Ofqual's Chief Regulator Glenys Stacey and Ofqual's Marking Consistency Metrics
4. TWO IMPORTANT WORDS: ‘ACCURATE’ AND ‘RELIABLE’. What does ‘accurate’ mean in the context of exams? Can exam marks ever be accurate? Looking at how marking by different examiners can alter the grade. The reliability of a grade is the probability that an originally-awarded grade is confirmed
5. SUMMER 2016: OFQUAL MAKE IT HARDER TO APPEAL. Until 2015, candidates unhappy with their grades could – for a fee – request a re-mark. But in May 2016, Ofqual changed the rules for challenges and appeals, intentionally denying access to an expert second opinion. Why did the regulator do that?
6. OFQUAL’S FIRST MEASURES OF GRADE RELIABILITY. In 2015, Ofqual carried out an extensive study in which the entire cohorts of GCSE, AS and A-level scripts, in 12 subjects, were marked twice: firstly, as normal, by an ordinary examiner; secondly, by a senior examiner, whose mark was designated 'true'
7. OFQUAL’S REAL MEASURES OF GRADE RELIABILITY. Using Ofqual data, the author calculates the reliability of mathematics grades is 96%, chemistry 92%, physics 88% etc right down to the lower arts subjects such as English literature (58%), history (56%) and combined English language and literature (52%)
8. WHY GRADES ARE UNRELIABLE. Three reasons why marking (marketing error) is not the problem. Instead, there is a more powerful explanation – fuzziness. Ofqual admit: 'There is often no single, correct mark for a question'. So marks may legitimately vary, causing valid but 'fuzzy' results
9. NOVEMBER 2018 TO SUMMER 2019. Newspaper sized on Ofqual's admission about grade unreliability, with reports in the Daily Telegraph and Sunday Times. Ofqual insists that marking error is to blame for any problems but stresses that results and grades are overwhelmingly accurate
10. 2020: CAGS AND RANK ORDERS. Due to the Covid epidemic, the UK government cancels all physical GCSE, AS-Level and A-Level exams and replaces them with an Ofqual algorithm, which is crude. 'The details of the algorithm were both important and missing from Ofqual’s Guidance documents.'
11. THE GREAT CAG CAR CRASH. On results day, Thursday 13th August, and over the next few days, progressively more stories surfaced on some ‘peculiarities’ in the algorithm’s results. Yes, overall, nearly 40% of CAGs were down-graded, but some were up-graded, sometimes very strangely
12. THE AFTERMATH. Were the CAGs right? Or fair? In a word, no. Or rather, some were, and some weren’t. I hope that most – if not the vast majority – were; but some definitely weren’t, and no one knows how many. Furthermore, Ofqual proved to be hugely obstructive in letting anyone find out
13. SUMMER 2021: THE TAGS. With Covid-19 once again obstructing physical exams, Ofqual oversaw Teachers Assessed Grades. 'The process was a mess in principle, and made even worse by the problems of rounding and statistics. But overall, Ofqual designed and implemented a totally flawed process...'
14. NINE WAYS TO DELIVER RELIABLE AND TRUSTWORTHY GRADES. Presents 14 solutions in order to offer a comprehensive ‘menu’ of possibilities, so as to stimulate the debate about which is indeed the best and most effective: the best and most effective among 15 possibilities
15. FIVE FUNDAMENTALLY DIFFERENT WAYS TO DELIVER RELIABLE AND TRUSTWORTHY ASSESSMENTS. Five more solutions to the problem of delivering reliable grades – or rather ‘assessments,’ for as will be seen, the last solution suggests a different way of representing students’ achievements on their certificates
16. OVER TO YOU… Reliable, trustworthy, grades are not just a public ‘good.’ Our young people deserve them. After 11, 12 and 13 years of school, the end-game, the key prizes, are those certificates with those grades. What a kick in the teeth it is for so many grades to be wrong
APPENDIX - FUZZINESS, A DEEPER DIVE. The central concept in this book is fuzziness – my word for the fact that different, equally qualified examiners can legitimately give the same script different marks. Here I define and describe fuzzy exam grades
ACKNOWLEDGEMENTS. Among others: Adi Bloom, Bernard Minsky, Bernard Trafford, Bill Watkin, Camilla Turner, Catherine Brioche, Carly Minsky, Elaine Hughes, Professor George Constantinides, Helen Pike, Dr Huy Duong, JL Dutaut, John Dickens, Liz Charin, Liz Lightfoot, Mark Corver, Mark Fretten
REFERENCES. A full list of statistical sources, newspaper articles, think tank reports, and regulatory information for this book on English school exam grades
INDEX. A full index. Such as the As: A-level 24 grade inflation Fig 5 (p 47) grade reliability, approximate measure for qualifications as awarded Fig 15, Table 3 (p 119) grade reliability, approximate measure for subject units or components Fig 13 (p 100), 104 results from ‘mutant algorithm’ 2020
| Erscheinungsdatum | 10.10.2022 |
|---|---|
| Vorwort | Dr Robin Bevan |
| Zusatzinfo | 15 black and white tables; 50 figures; 1 iondex |
| Sprache | englisch |
| Maße | 156 x 234 mm |
| Gewicht | 650 g |
| Themenwelt | Sozialwissenschaften ► Pädagogik |
| ISBN-10 | 1-912454-99-8 / 1912454998 |
| ISBN-13 | 978-1-912454-99-0 / 9781912454990 |
| Zustand | Neuware |
| Haben Sie eine Frage zum Produkt? |
aus dem Bereich
