This is a follow-up to my earlier post on HEFCE’s Quality Assessment consultation. This post elaborates the reasons why using student outcomes data to assess educational quality is unacceptable, in the form of an open letter that all academic colleagues are invited to sign. To do so, just add your name, title and institutional affiliation in the comments.

UPDATE 23.09.15: UUK has just circulated their response to HEFCE, which endorses the use of student outcomes data, making this letter all the more urgent and necessary. They write:

We agree that a core set of quantitative student outcome metrics should be included in institutional reporting. These should be the benchmarked UK performance indicator set, covering retention, widening participation, 6 months destination of leavers from higher education, plus relevant benchmarked results from the national student survey, primarily question 22 ‘overall satisfaction with course’…

*    *    *

To: HEFCE and Universities UK

We are university educators who are deeply disturbed by the proposed use of ‘student outcomes’ metrics as proxy measures of teaching quality in HEFCE’s proposed new ‘quality assessment’ regime and the mooted ‘Teaching Excellence Framework’. We call upon HEFCE to reject, and upon UUK to campaign vociferously against, this use.

The use of ‘student outcomes’ to measure teaching quality is completely inappropriate for the following reasons.

1. Student attainment is never a direct or reliable measure of teaching quality because it is influenced by a host of factors unconnected to the quality of teaching. Decades of research has shown that the largest single determinant of educational outcomes is social class.^[1] Yet, in secondary education, the OFSTED inspection regime and much political/ public discourse operates as if ‘good teaching’ is the only important driver, and by extension bad outcomes must be the result of ‘bad teaching’. This is simply inaccurate. Similarly, in Higher Education, student attainment within single academic departments frequently fluctuates considerably year-on-year. This does not necessarily signify fluctuations in teaching quality, which experience suggests remains broadly consistent, but rather cohort effects based on the characteristics of a given student body – not least the students’ own efforts. The same is true for many other metrics, such as student employment and salary data, which are heavily influenced by many non-teaching factors, notably interpersonal networks, which in turn often rest on social class themselves.^[2] Similar concerns have been raised about the use of NSS data.^[3] Given the strong social determinants of student outcomes, any metrics system based on them may reflect pre-existing social hierarchies rather than providing any objective measure of ‘teaching quality’.
2. The clear risk in using ‘student outcomes’ as a measurement of teaching quality is that it removes responsibility for outcomes from the students themselves. At root, higher education is not a ‘service’ that HEIs ‘deliver’. Higher Education is a relationship between a teacher and a student, who both bring something crucial to the table: the teacher teaches, but it is the adult student who learns, or does not learn. Of course, some students struggle to learn as a result of inequalities in prior educational experiences, the failure to make reasonable adjustments to student needs, or poor teaching practice. These clearly need to be addressed by HEIs and there are various (broadly successful) systems in place for doing so. However, other reasons for failure are rightly understood to be the responsibility of students themselves. Learning is intrinsically difficult, and UK HE has rightly and historically insisted that university education is about students undertaking guided but independent study. However, students obviously vary in their motivation and capacity to struggle to acquire new skills and knowledge. If student outcomes are implicitly made the sole responsibility of teachers, the message sent to students is that they do not need to take ownership of their own learning experience and work hard to succeed: if they fail, it is the fault of their lecturers. It would advance a lamentable tendency whereby students are treated as mere ‘consumers’ of HE who can expect a particular outcome, rather than as independent learners who must strive to take advantage of all the educational opportunities open to them and be accountable for their own progress. We must not remove responsibility from young adults for learning for themselves, managing their own progress and meeting expectations.
3. When a focus on outcomes is coupled with a demand for ‘continuous improvement’ (Future Approaches to Quality, paragraph 41(e)), this is an obvious recipe for grade inflation, despite HEFCE’s stated wish to tackle this problem (paragraph 79). This would only further invalidate the use of student attainment as a metric of good teaching, because the gap between a student’s qualifications at entry and exit is not a reliable indicator of the intellectual distance travelled. So-called ‘value added’ could only be measured (if at all) by holding purpose-made exams for students at entry and exit. Again this would undermine HEI autonomy, innovation and experimentation – and indeed educational quality – by encouraging teaching to the test, as is now endemic in UK secondary education.
4. As is well understood in the literature on pedagogy and the traditions of UK HE, higher education is not simply about producing students with degrees but more fundamentally about promoting a range of lifelong skills and qualities within students, which are difficult if not impossible to capture as ‘student outcomes. These include including independent critical thinking, self-organisation, community service, knowledge of national and international processes and developments and so on. By fixing assessments of teaching quality to measurable numerical outcomes, HEIs and teachers will be strongly incentivised to spend less time and effort on these wider social goods and skills for their students. This will lead to a narrowing of the pedagogic mission within public universities.
5. Finally, there are particular dangers relating to what specific indicators are taken as ‘student outcomes’. Indicators such as graduate earnings have been suggested, but these are highly inappropriate as proxies for teaching quality in a public higher education system. Comparing the salaries of, say, primary school teachers and commercial lawyers says virtually nothing about the quality of the teaching which led to their employment, nor their respective contributions to society, and could potentially de-value subjects and disciplines where students are oriented towards less remunerative but socially important professions. We have already seen moves in this direction with the introduction of Key Information Sets, and the ‘Future Employment and Earnings Record’ mooted by the 2014 Lord Young Review of Enterprise in Education. In these metrics, the foregrounding of average earnings from different degrees are intended to steer students towards remuneration rather than societal contribution. Lamentably, this has been coupled with the active discouragement of students from undertaking Arts degrees by the Education Secretary, Nicky Morgan.

[1] The evidence base is enormous. Some of the most influential works include: J.A. Goldthorpe, ‘Class Analysis and the Reorientation of Class Theory: The Case of Persisting Differentials in Educational Attainment’, British Journal of Sociology 47, 3 (1996): 481-505; Annette Lareau, Home Advantage: Social Class and Parental Intervention in Elementary Education (Lanham: Rowman and Littlefield, 2000); Alice Sullivan, ‘Cultural Capital and Educational Attainment’, Sociology 35, 4 (2001): 893-912.

[2] See the classic study, Mark Granovetter, Getting a Job: A Study of Contacts and Careers (Chicago: University of Chicago Press, 1974/1995) and, most recently, Social Mobility and Child Poverty Commission, A Qualitative Evaluation of Non-Educational Barriers to the Elite Professions, June 2015; available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/434791/A_qualitative_evaluation_of_non-educational_barriers_to_the_elite_professions.pdf. For a general discussion of why graduate destinations cannot serve as a proxy measure for teaching quality, see https://academicirregularities.wordpress.com/2015/07/28/how-not-to-measure-teaching-quality-and-learning-gain/.

[3] John Holmwood, ‘Code of Practice Needed to Prevent Degree-Course Mis-selling’, Research Blogs, 7 February 2011; available at http://exquisitelife.researchresearch.com/exquisite_life/2011/02/code-of-practice-needed-to-halt-degree-course-mis-selling-.html.