Can Metrics Be Used Responsibly? Why Structural Conditions Push Against This

Not waving, exactly, but...

Not waving, exactly, but…

Today, the long-awaited ‘Metric Tide’ report from the Independent Review of the Role of Metrics in Research Assessment and Management was published, along with appendices detailing its literature review and correlation studies. The main take-away is: IF you’re going to use metrics, you should use them responsibly (NB NOT: You should use metrics and use them responsibly). The findings and ethos are covered in the Times Higher and summarised in Nature by the Review Chair James Wilsdon, and further comments from Stephen Curry (Review team) and Steven Hill (HEFCE) are published. I highly recommend this response to the findings by David Colquhoun. You can also follow #HEFCEMetrics on Twitter for more snippets of the day. Comments by Cambridge lab head Professor Ottoline Leyser were a particular highlight.

I was asked to give a response to the report at today’s launch event, following up on the significance of mine and Pablo’s widely endorsed submission to the review. I am told that the event was recorded by video and audio so I will add links to that when they show up. But before then, a short summary record of the main points I made: Continue reading

Metrics: An Addendum on RAE / REF

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading

Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to metrics@hefce.ac.uk). If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at]soas.ac.uk or p.c.kirby[at]sussex.ac.uk before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.


Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex

Summary

Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading

Getting Somewhere: HEFCE Proposals on Open Access for a Post-2014 Research Excellence Framework

zimbabwe-press

This week, the UK’s Higher Education Funding Council for England (HEFCE) published their formal proposals for including an open access requirement in any post-2014 Research Excellence Framework (REF). Responses to this will be accepted until 30th October 2013. These proposals follow a pre-consultation letter and set of responses which were submitted earlier in the year (link to University of Cambridge response).

Following up our concerns about the policy raised over the last few months (here and here, further posts here) the present iteration represents a decent outcome on some of the details, not least because it defers quite a few of them. That these issues have been deferred does not mean that they do not matter; rather it means that the battles on them will be fought elsewhere – with universities, with journal boards, with learned societies, with publishers and their lawyers and so on. Moreover, there is no cause for complacency around the broader political economy of scholarly publishing, which remains wasteful, restrictive and inequitable on many fronts. And of course, the pernicious REF exercise itself, which this government signalled it would review, must be itself vigorously contested (more on this to come).

The Requirements

The proposals are to require that any REF-submitted journal article or conference proceeding published after 2016 must be made available in the final post-peer reviewed version from an institutional repository at the point of acceptance (or publication). This in line with the previous agenda of RCUK and others, and maintains journal exclusivity by accepting substantial embargo times on truly open (read: public) viewing of these deposited versions. So a paper “immediately” placed in an institutional repository may still not be viewable for up to 24 months in the Arts, Humanities and Social Sciences (recall that this length of permissible embargo was extended in the face of publisher lobbying), although it may be possible to request papers directly from the author on an individual basis. This version of mandated open access is applicable only if the address line of the author is a UK HEI at the time of publication, and cases for exceptions can be made. HEFCE’s assessment that this should be broadly achievable under current provisions is reasonable, and represents a good path for opening up access to research under present conditions. From the perspective of maximising green OA, Stevan Harnad’s response is as ever highly incisive. Below are some reflections on the present state of play.

What Has Been Won and Deferred

First, the old green/gold battle is now (nearly) over, and with it the concerns that academics would routinely pay exorbitant author fees to have their research published. Continue reading

Open Access: HEFCE, REF2020 and the Threat to Academic Freedom

This is the text of a document prepared by Meera and me on Article Processing Charges as currently understood and the serious risks we think they pose to academic freedom and funding, broadly understood (previous discussed by several contributors to our open access series). It is also available as a pdf, and we encourage academics to think carefully about the issues foregrounded, and to act accordingly.


Applegarth Press

Summary

  • The Government is pushing academic publishing to a ‘pay-to-say’ model in order to achieve open access to publicly funded research
  • This ‘gold’ route to open access, which levies Article Processing Charges (as proposed in the Finch Report and taken up by RCUK and HEFCE) poses a major problem for academics in the UK:
    • It threatens academic freedom through pressures on institutions to distribute scarce APC resources and to judge work by standards other than peer review
    • It threatens research funding by diverting existing funds into paying for publications (and private journal profits) rather than into research
    • It increases academic inequality both across and within institutions, by linking prestige in research and publishing to the capacity to pay APCs, rather than to academic qualities
    • It threatens academic control of research outputs by allowing for commercial uses without author consent
  • In response, academics should:
    • Practice and lobby for ‘green’ open access of all post-peer reviewed work within journals and institutions
    • Lobby against proposed restrictions on REF2020 and against compliance pressure for ‘gold’ open access
    • Demand clear policies from Universities around open access funds
    • Ensure institutional resources are not unnecessarily spent on APCs
    • Protect the integrity of scholarly journals by rejecting the pressure for ‘pay-to-say’ publishing

Open Access: Rushing Implementation

Many academics have been ardent supporters of the open access principle (that peer-reviewed academic work should be freely available and easily accessible to anyone), and were excited when the Government made steps to advance it. However, it has become clear that the implementation of this policy via REF2020 will have very serious negative consequences for all academic authors and institutions, unless authors and institutions themselves start to take action and make their voices heard. It is critical that academics understand what is happening and lobby our pro-VCs of Research, our VCs and Universities UK to defend both academic freedom and open access.

The timescale for action and decision-making is now incredibly short. Several policies, including that of the Government and of RCUK were declared immediately with the release of the Finch Report, totally accepting its views without wider consultation. HEFCE is going to open and close a very quick consultation period early in 2013 in order to issue guidance ahead of REF2020. Some universities have been given until March 2013 to determine what to do with open access funds that they were given in November. And it was only on 29 November 2012 that the first indications from HEFCE were given as to their intentions, at the Academy of Social Sciences (ACSS) conference on Implementing Finch. The timetable for finalising the details of this complex policy is thus extremely short and does not allow for adequate discussion of its serious consequences. Despite this, academics can still play an important role in resisting the threats posed.

So, What is Happening?

In summary, academic journals are being moved from a ‘pay-to-read’ model to a ‘pay-to-say’ model.

Continue reading

Open Access: News and Reflections from the ACSS Conference

Last week, the first big public event discussing the Open Access policy announced in July was held at the Royal Statistical Society by the Academy of Social Sciences. If you are interested, many of the presentations from the event are already available online, with more write-ups to follow, as well as a promised YouTube video of the entire event. The programme promised and delivered a good range of speakers, and not least Dame Janet Finch herself.  I went along for the first day, thinking that this might be an open space to learn about the issues and have discussions about the policy, involving a wide range of affected parties.

Janet Finch

I did learn a lot, although what I mainly learned was that no one was really prepared to take any real responsibility for a policy to which a lot of eminent and well-informed people had very serious objections. Finch insisted that she had to stick to a brief which did not involve ‘destabilising’ the publishing system. No one was there to answer from either BIS or RCUK, who both adopted the policy immediately upon the publication of the report in July. HEFCE, who have not formally announced a position yet, however indicated at the conference that they are very likely to adopt the RCUK model for REF2020. Continue reading

One Size Fits All?: Social Science and Open Access

The third post in our small series on open access, publication shifts on the horizon and how it all matters to IR and social science, this time by David Mainwaring  (Pablo’s post was firstthen Colin Wight’s, and following David came Nivi Manchanda, Nathan Coombs and our own Meera). David is a Senior Editor at SAGE with responsibility for journals in politics and international studies. So he oversees journals like Millennium and European Journal of International Relations amongst others (which is how we know him), and can thus offer a close reading of movements within the academic publishing industry. Images by Pablo.


Open Access is the talk of the academic town. The removal of barriers to the online access and re-use of scholarly research is being driven by a cluster of technological, financial, moral and commercial imperatives, and the message from governments and funding agencies is clear: the future is open. What is much less clear is exactly what sort of open future social scientists would benefit from, let alone what steps need to be taken in order to transition away from the existing arrangements of scholarly communication and validation. Here the conversation is in its relative infancy, characterised at this point by a great deal of curiosity, anticipation, confusion, and the shock of the new. What it needs to move towards is a recognition and coordinated response to the fact that although social science may share the same open access goal as the STEM disciplines, the motivations for travelling down that path are not identical, and the context – especially in terms of research funding – is significantly different. The roundtable discussion at the Millennium conference at the LSE on 20th October was an attempt to explore these issues specifically from an IR perspective; further such events (such as those being run by the AcSS and LSE this autumn) are to be warmly welcomed as a means of building broader understanding of the issues among social scientists and facilitating strategic thinking.

Social Science and the Open Access Debate

The long-running debate about how scholarly research communication should be funded and transmitted has been, and remains, a discussion conducted primarily by those working in STEM. Most of the blogosphere’s best-known voices on open access, the likes of Mike Taylor, Michael Eisen, Peter Murray-Rust, Björn Brembs, Cameron Neylon, Kent Anderson, Stephen Curry and Tim Gowers, have backgrounds in STEM research or publishing. (Notable exceptions are the philosopher Peter Suber, who heads the Harvard Open Access Project, and self-archiving advocate Stevan Harnad, a cognitive scientist). To dip into the often heated debates on open access can leave you with the strong impression that, despite the occasional nod to social science and the humanities, the frame of reference is proper, rigorous, natural scientific research, the kind carried out in a laboratory that leads to medical advances and the development of new technologies.

By and large social scientists – and arts and humanities scholars, to whom many of the points raised in this piece apply equally – have had a back seat in this conversation, and the development of open access awareness and capabilities has been slow. Many leading social science journals continue to be distributed in print form due to subscriber demand well over a decade after the launch of their online editions. The American Political Science Association’s 2009 book Publishing Political Science devotes just two out of over 250 pages to open access, and fewer than 10% of the nearly 13,000 signatories of the ‘Cost of Knowledge‘ boycott of Elsevier were social scientists, despite the company’s position as one of the world’s leading publishers of social science journals. That’s not to say that there is nothing going on: the Social Science Research Network has acted as a site for open paper sharing since 1994; there are active ‘open’ movements in disciplines including IR and economics; and the Directory of Open Access Journals lists more than 1,600 social science titles. To date, however, very few of the latter have been able to break into the higher echelons of profile or reputation within their fields.

Social Science and Open Access Mandates

Over the last eighteen months a series of events, including George Monbiot’s polemic in The Guardian and the defeat in the US of the Research Works Act, stirred for the first time a significant consciousness among social scientists about open access. This awareness increased dramatically – in the UK at least – with the publication in June 2012 of the final report of a committee set up by the Government to examine access to published research findings. Continue reading