This is a follow-up to my earlier post on HEFCE’s Quality Assessment consultation. This post elaborates the reasons why using student outcomes data to assess educational quality is unacceptable, in the form of an open letter that all academic colleagues are invited to sign. To do so, just add your name, title and institutional affiliation in the comments.
UPDATE 23.09.15: UUK has just circulated their response to HEFCE, which endorses the use of student outcomes data, making this letter all the more urgent and necessary. They write:
We agree that a core set of quantitative student outcome metrics should be included in institutional reporting. These should be the benchmarked UK performance indicator set, covering retention, widening participation, 6 months destination of leavers from higher education, plus relevant benchmarked results from the national student survey, primarily question 22 ‘overall satisfaction with course’…
* * *
To: HEFCE and Universities UK
We are university educators who are deeply disturbed by the proposed use of ‘student outcomes’ metrics as proxy measures of teaching quality in HEFCE’s proposed new ‘quality assessment’ regime and the mooted ‘Teaching Excellence Framework’. We call upon HEFCE to reject, and upon UUK to campaign vociferously against, this use.
The use of ‘student outcomes’ to measure teaching quality is completely inappropriate for the following reasons.
- Student attainment is never a direct or reliable measure of teaching quality because it is influenced by a host of factors unconnected to the quality of teaching. Decades of research has shown that the largest single determinant of educational outcomes is social class.[1] Yet, in secondary education, the OFSTED inspection regime and much political/ public discourse operates as if ‘good teaching’ is the only important driver, and by extension bad outcomes must be the result of ‘bad teaching’. This is simply inaccurate. Similarly, in Higher Education, student attainment within single academic departments frequently fluctuates considerably year-on-year. This does not necessarily signify fluctuations in teaching quality, which experience suggests remains broadly consistent, but rather cohort effects based on the characteristics of a given student body – not least the students’ own efforts. The same is true for many other metrics, such as student employment and salary data, which are heavily influenced by many non-teaching factors, notably interpersonal networks, which in turn often rest on social class themselves.[2] Similar concerns have been raised about the use of NSS data.[3] Given the strong social determinants of student outcomes, any metrics system based on them may reflect pre-existing social hierarchies rather than providing any objective measure of ‘teaching quality’.
- The clear risk in using ‘student outcomes’ as a measurement of teaching quality is that it removes responsibility for outcomes from the students themselves. At root, higher education is not a ‘service’ that HEIs ‘deliver’. Higher Education is a relationship between a teacher and a student, who both bring something crucial to the table: the teacher teaches, but it is the adult student who learns, or does not learn. Of course, some students struggle to learn as a result of inequalities in prior educational experiences, the failure to make reasonable adjustments to student needs, or poor teaching practice. These clearly need to be addressed by HEIs and there are various (broadly successful) systems in place for doing so. However, other reasons for failure are rightly understood to be the responsibility of students themselves. Learning is intrinsically difficult, and UK HE has rightly and historically insisted that university education is about students undertaking guided but independent study. However, students obviously vary in their motivation and capacity to struggle to acquire new skills and knowledge. If student outcomes are implicitly made the sole responsibility of teachers, the message sent to students is that they do not need to take ownership of their own learning experience and work hard to succeed: if they fail, it is the fault of their lecturers. It would advance a lamentable tendency whereby students are treated as mere ‘consumers’ of HE who can expect a particular outcome, rather than as independent learners who must strive to take advantage of all the educational opportunities open to them and be accountable for their own progress. We must not remove responsibility from young adults for learning for themselves, managing their own progress and meeting expectations.
- When a focus on outcomes is coupled with a demand for ‘continuous improvement’ (Future Approaches to Quality, paragraph 41(e)), this is an obvious recipe for grade inflation, despite HEFCE’s stated wish to tackle this problem (paragraph 79). This would only further invalidate the use of student attainment as a metric of good teaching, because the gap between a student’s qualifications at entry and exit is not a reliable indicator of the intellectual distance travelled. So-called ‘value added’ could only be measured (if at all) by holding purpose-made exams for students at entry and exit. Again this would undermine HEI autonomy, innovation and experimentation – and indeed educational quality – by encouraging teaching to the test, as is now endemic in UK secondary education.
- As is well understood in the literature on pedagogy and the traditions of UK HE, higher education is not simply about producing students with degrees but more fundamentally about promoting a range of lifelong skills and qualities within students, which are difficult if not impossible to capture as ‘student outcomes. These include including independent critical thinking, self-organisation, community service, knowledge of national and international processes and developments and so on. By fixing assessments of teaching quality to measurable numerical outcomes, HEIs and teachers will be strongly incentivised to spend less time and effort on these wider social goods and skills for their students. This will lead to a narrowing of the pedagogic mission within public universities.
- Finally, there are particular dangers relating to what specific indicators are taken as ‘student outcomes’. Indicators such as graduate earnings have been suggested, but these are highly inappropriate as proxies for teaching quality in a public higher education system. Comparing the salaries of, say, primary school teachers and commercial lawyers says virtually nothing about the quality of the teaching which led to their employment, nor their respective contributions to society, and could potentially de-value subjects and disciplines where students are oriented towards less remunerative but socially important professions. We have already seen moves in this direction with the introduction of Key Information Sets, and the ‘Future Employment and Earnings Record’ mooted by the 2014 Lord Young Review of Enterprise in Education. In these metrics, the foregrounding of average earnings from different degrees are intended to steer students towards remuneration rather than societal contribution. Lamentably, this has been coupled with the active discouragement of students from undertaking Arts degrees by the Education Secretary, Nicky Morgan.
[1] The evidence base is enormous. Some of the most influential works include: J.A. Goldthorpe, ‘Class Analysis and the Reorientation of Class Theory: The Case of Persisting Differentials in Educational Attainment’, British Journal of Sociology 47, 3 (1996): 481-505; Annette Lareau, Home Advantage: Social Class and Parental Intervention in Elementary Education (Lanham: Rowman and Littlefield, 2000); Alice Sullivan, ‘Cultural Capital and Educational Attainment’, Sociology 35, 4 (2001): 893-912.
[2] See the classic study, Mark Granovetter, Getting a Job: A Study of Contacts and Careers (Chicago: University of Chicago Press, 1974/1995) and, most recently, Social Mobility and Child Poverty Commission, A Qualitative Evaluation of Non-Educational Barriers to the Elite Professions, June 2015; available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/434791/A_qualitative_evaluation_of_non-educational_barriers_to_the_elite_professions.pdf. For a general discussion of why graduate destinations cannot serve as a proxy measure for teaching quality, see https://academicirregularities.wordpress.com/2015/07/28/how-not-to-measure-teaching-quality-and-learning-gain/.
[3] John Holmwood, ‘Code of Practice Needed to Prevent Degree-Course Mis-selling’, Research Blogs, 7 February 2011; available at http://exquisitelife.researchresearch.com/exquisite_life/2011/02/code-of-practice-needed-to-halt-degree-course-mis-selling-.html.
Pingback: ‘Quality Assessment’ and Completing the Market in UK Higher Education | The Disorder Of Things
Meera Sabaratnam, SOAS
LikeLike
Elisa Wynne-Hughes, Cardiff University
LikeLike
Andy Panay, Abertay University
LikeLike
John McTague, University of Bristol
LikeLike
Director of Learning and Teaching, Social Sciences and Humanities, University of Westminster
LikeLike
Thomas Moore 🙂
LikeLike
Chris Rossdale, University of Warwick
LikeLike
Joe Hoover, City University London
LikeLike
Peter Brett, Queen Mary University of London
LikeLike
David Webber, University of Warwick
LikeLike
Reblogged this on oshriradhekrishnabole.
LikeLike
Rahul Rao, SOAS
LikeLike
Helen Dexter, University of Leicester
LikeLike
Totally agree.
LikeLike
Jonathan Gilmore, Kingston University
LikeLike
Hannah Cross
LikeLike
Laleh Khalili, SOAS
LikeLike
Richard Hatcher
Professor of Education
Birmingham City University
LikeLike
Alessandra Mezzadri, SOAS
LikeLike
Lori Allen, SOAS
LikeLike
Sneha Krishnan, University of Oxford
LikeLike
Manjeet Ramgotra, SOAS
LikeLike
Anthony Faramelli, Kingston University
LikeLike
Catherine Goetze. University of Sussex
LikeLike
Tony Barlow, Birmingham City University
LikeLike
Maia Pal, Lecturer IR, Oxford Brookes University
LikeLike
Saladin Meckled-Garcia, University College London
LikeLike
cynthia weber
university of sussex
LikeLike
Adam Morton,
University of Sydney
LikeLike
John Filling, University of Cambridge
LikeLike
Sarah Bulmer, University of Exeter
LikeLike
Tara McCormack, University of Leicester
LikeLike
Robert Cowley, University of Westminster and King’s College London
LikeLike
Jason Scott-Warren, Faculty of English, University of Cambridge
LikeLike
Lara Montesinos Coleman, Lecturer in International Relations and International Development, University of Sussex
LikeLike
Devon Curtis, University of Cambridge
LikeLike
Andrew Slack, Manchester University
LikeLike
Yuliya Yurchenko, University of Greenwich
LikeLike
PGCE (post compulsory art and design) Northbrook College, Sussex
LikeLike
Daniel Conway, University of Westminster
LikeLike
Owen Miller, SOAS, University of London
LikeLike
Antoine Bousquet, Birkbeck College
LikeLike
Prof Laurie Stras
University of Southampton
LikeLike
Professor Deborah Lynn Steinberg
University of Warwick
LikeLike
Debbie Epstein, Professor of Cultural Studies in Education, University of Roehampton
LikeLike
Chris Jones, Professor of Research in Educational Technology, Liverpool John Moores University
LikeLike
Dr Barbara Eichner, Senior Lecturer in Music, Oxford Brookes University
LikeLike
Angela Fenwick, Faculty of Medicine, University of Southampton
LikeLike
Bangor University (until 30 September)
University of Oxford (from 1 October)
LikeLike
Professor Thomas Schmidt
University of Manchester
LikeLike
I wholeheartedly agree with this open letter.
LikeLike
Dr Muir Houston
Senior Lecturer, School of Eduction, University of Glasgow
LikeLike
Dr Richie Nimmo,
Lecturer in Sociology, University of Manchester
LikeLike
Dr Jerry Goodenough, Senior Lecturer in Philosophy, University of East Anglia
LikeLike
Dr Dawn Lyon, University of Kent
LikeLike
Dr Chris Maunder, York St John University
LikeLike
Please add my name, I completely support
Gregor McLennan
Professor of Sociology & Head of School,
Sociology, Politics and International Studies
University of Bristol
LikeLiked by 1 person
In full support
LikeLike
Dr. Nadia Edmond University of Brighton School of Education
LikeLike
Dr Stuart Jesson
Lecturer in Religion, Philosophy and Ethics
York St John University
LikeLike
Dr Hannah Holtschneider
Senior Lecturer in Jewish Studies
University of Edinburgh
LikeLike
Kerry Harman,
Programme Director, Higher Education Introductory Studies
Birkbeck
LikeLike
Sean Sayers
Emeritus Professor of Philosophy
University of Kent
LikeLike
Andrew Goffey
Director, Centre for Critical Theory
University of Nottingham
LikeLike
Dr Lynne Pettinger, University of Warwick
LikeLike
Dr. Sherrill Stroschein, University College London
LikeLike
Jo Frankham
Liverpool John Moores University
Research (funded by the British Academy/Leverhulme) on the punitive and negative effects of the NSS, just published, available from me at: j.frankham@ljmu.ac.uk
LikeLike
Kate Tunstall, University of Oxford
LikeLike
Dr Paul Kirby, University of Sussex
LikeLike
Dr Marika Rose, Research Fellow, Durham University.
LikeLike
Dr Aggie Hirst, City University London
LikeLike
Dr Jeppe Mulich, LSE
LikeLike
Adam Humphreys, University of Reading
LikeLike
Kathryn Telling, University of Manchester
LikeLike
Nicholas Kitchen
London School of Economics
LikeLike
Fran Tracy, Lecturer in Education Studies at Liverpool John Moores University
LikeLike
Jonathan Rodwell, Manchester Metropolitan University
LikeLike
The letter was sent on 12 October and so this post is now closed. If you want to comment on the substantive issues raised please do so at the initial post on QA here: https://thedisorderofthings.com/2015/09/16/quality-assessment-and-completing-the-market-in-uk-higher-education/
LikeLike
Pingback: HE Green Paper and related stuff | Richard Hall's Space