Can Metrics Be Used Responsibly? Why Structural Conditions Push Against This

Not waving, exactly, but...

Not waving, exactly, but…

Today, the long-awaited ‘Metric Tide’ report from the Independent Review of the Role of Metrics in Research Assessment and Management was published, along with appendices detailing its literature review and correlation studies. The main take-away is: IF you’re going to use metrics, you should use them responsibly (NB NOT: You should use metrics and use them responsibly). The findings and ethos are covered in the Times Higher and summarised in Nature by the Review Chair James Wilsdon, and further comments from Stephen Curry (Review team) and Steven Hill (HEFCE) are published. I highly recommend this response to the findings by David Colquhoun. You can also follow #HEFCEMetrics on Twitter for more snippets of the day. Comments by Cambridge lab head Professor Ottoline Leyser were a particular highlight.

I was asked to give a response to the report at today’s launch event, following up on the significance of mine and Pablo’s widely endorsed submission to the review. I am told that the event was recorded by video and audio so I will add links to that when they show up. But before then, a short summary record of the main points I made: Continue reading

Metrics: An Addendum on RAE / REF

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading

Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at] or p.c.kirby[at] before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.

Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex


Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading

HEFCE, the REF, Open Access and Journals in Politics, IR and Political Theory

UPDATE (8 September 2014): Lee originally wrote this as a guest post – providing some much needed concrete detail on journal open access policies – but is now with us for good.

Following the launch of HEFCE’s consultation on Open Access for any post-2014 REF, and the generally positive reaction to it here, I examined the potential implications of HEFCE’s proposals for journal publishing in Politics, IR and Political Theory. I wanted to know whether the serious threat in HEFCE’s earlier proposals – notably the rush for ‘gold’ OA and associated Article Processing Charges (APCs) – had been eliminated by a downgrading of the proposals to permit ‘green’ OA, by depositing a ‘final accepted version’ (FAV, i.e. post-peer review, but pre-type setting) into an institutional repository. I also wanted to see what embargoes journals placed on FAVs (i.e. how long after the ‘Version of Record’ (VoR) is published in the journal the FAV can be made public); what re-use was permitted (what sort of licensing); and also to compare this route with the ‘gold’ access favoured by the Finch Report and the RCUK policy. I also wanted to gather information that could be used as part of a ‘soft boycott’ of OA-resistant outlets.

To do this I selected 57 journals that broadly represent the ‘top’ journals in the three subfields. I used composite rankings from Google Scholar, the ISI Citations Index, and surveys of scholars where available, and got feedback on an initial draft. The list isn’t intended to be definitive, just to give us a better sense of where the ‘big journals’ in which many scholars aspire to publish actually stand on OA.

It is not easy to get this information. Policies can vary by journal, not merely by publisher, and their websites are often opaque on the issue of self-archiving, particularly in terms of licensing. This may change if, as seems likely, HEFCE forges ahead on OA; but publishers also need to be pushed to display clearer information. The exaggerated nature of the Finch Report’s estimate of UK HE’s market power to change publishing models is underscored by the fact that US journals tend not to provide the information I was looking for (my thanks to Sarah Molloy for help with this).

The results are presented below for Politics, IR and Political Theory (click to enlarge each image). There are a lot of complexities with various journals which are shown in the full spreadsheet of results which you can see here; the spreadsheet also lets you reorder the information by different criteria.


Journal Open Access Policies for International Relations.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Political Theory.

Journal Open Access Policies for Political Theory.

Several conclusions can be drawn.

(1) Most importantly, Pablo and Meera were basically right: the threat of HEFCE rushing to a ‘pay to say’ approach with hugely detrimental financial consequences for universities and the potential for internal rationing of APCs, has been defeated, for now (although RCUK’s policies remain intact). Some journals, particularly in the US, don’t seem to support green FAV self-archiving, or have gold APC routes – this is a problem because they would be non-compliant with HEFCE’s proposals; so we will need to push for exemptions on this score. However, by and large, where they allow FAV self-archiving, none of the journals appear to charge a fee; HEFCE’s current proposals now intend to compel more researchers to do what they are already entitled to do. So long as this does not change, HEFCE’s proposals should not involve ruinous costs or lead to APC rationing.

Continue reading

Getting Somewhere: HEFCE Proposals on Open Access for a Post-2014 Research Excellence Framework


This week, the UK’s Higher Education Funding Council for England (HEFCE) published their formal proposals for including an open access requirement in any post-2014 Research Excellence Framework (REF). Responses to this will be accepted until 30th October 2013. These proposals follow a pre-consultation letter and set of responses which were submitted earlier in the year (link to University of Cambridge response).

Following up our concerns about the policy raised over the last few months (here and here, further posts here) the present iteration represents a decent outcome on some of the details, not least because it defers quite a few of them. That these issues have been deferred does not mean that they do not matter; rather it means that the battles on them will be fought elsewhere – with universities, with journal boards, with learned societies, with publishers and their lawyers and so on. Moreover, there is no cause for complacency around the broader political economy of scholarly publishing, which remains wasteful, restrictive and inequitable on many fronts. And of course, the pernicious REF exercise itself, which this government signalled it would review, must be itself vigorously contested (more on this to come).

The Requirements

The proposals are to require that any REF-submitted journal article or conference proceeding published after 2016 must be made available in the final post-peer reviewed version from an institutional repository at the point of acceptance (or publication). This in line with the previous agenda of RCUK and others, and maintains journal exclusivity by accepting substantial embargo times on truly open (read: public) viewing of these deposited versions. So a paper “immediately” placed in an institutional repository may still not be viewable for up to 24 months in the Arts, Humanities and Social Sciences (recall that this length of permissible embargo was extended in the face of publisher lobbying), although it may be possible to request papers directly from the author on an individual basis. This version of mandated open access is applicable only if the address line of the author is a UK HEI at the time of publication, and cases for exceptions can be made. HEFCE’s assessment that this should be broadly achievable under current provisions is reasonable, and represents a good path for opening up access to research under present conditions. From the perspective of maximising green OA, Stevan Harnad’s response is as ever highly incisive. Below are some reflections on the present state of play.

What Has Been Won and Deferred

First, the old green/gold battle is now (nearly) over, and with it the concerns that academics would routinely pay exorbitant author fees to have their research published. Continue reading

Open Access: HEFCE, REF2020 and the Threat to Academic Freedom

This is the text of a document prepared by Meera and me on Article Processing Charges as currently understood and the serious risks we think they pose to academic freedom and funding, broadly understood (previous discussed by several contributors to our open access series). It is also available as a pdf, and we encourage academics to think carefully about the issues foregrounded, and to act accordingly.

Applegarth Press


  • The Government is pushing academic publishing to a ‘pay-to-say’ model in order to achieve open access to publicly funded research
  • This ‘gold’ route to open access, which levies Article Processing Charges (as proposed in the Finch Report and taken up by RCUK and HEFCE) poses a major problem for academics in the UK:
    • It threatens academic freedom through pressures on institutions to distribute scarce APC resources and to judge work by standards other than peer review
    • It threatens research funding by diverting existing funds into paying for publications (and private journal profits) rather than into research
    • It increases academic inequality both across and within institutions, by linking prestige in research and publishing to the capacity to pay APCs, rather than to academic qualities
    • It threatens academic control of research outputs by allowing for commercial uses without author consent
  • In response, academics should:
    • Practice and lobby for ‘green’ open access of all post-peer reviewed work within journals and institutions
    • Lobby against proposed restrictions on REF2020 and against compliance pressure for ‘gold’ open access
    • Demand clear policies from Universities around open access funds
    • Ensure institutional resources are not unnecessarily spent on APCs
    • Protect the integrity of scholarly journals by rejecting the pressure for ‘pay-to-say’ publishing

Open Access: Rushing Implementation

Many academics have been ardent supporters of the open access principle (that peer-reviewed academic work should be freely available and easily accessible to anyone), and were excited when the Government made steps to advance it. However, it has become clear that the implementation of this policy via REF2020 will have very serious negative consequences for all academic authors and institutions, unless authors and institutions themselves start to take action and make their voices heard. It is critical that academics understand what is happening and lobby our pro-VCs of Research, our VCs and Universities UK to defend both academic freedom and open access.

The timescale for action and decision-making is now incredibly short. Several policies, including that of the Government and of RCUK were declared immediately with the release of the Finch Report, totally accepting its views without wider consultation. HEFCE is going to open and close a very quick consultation period early in 2013 in order to issue guidance ahead of REF2020. Some universities have been given until March 2013 to determine what to do with open access funds that they were given in November. And it was only on 29 November 2012 that the first indications from HEFCE were given as to their intentions, at the Academy of Social Sciences (ACSS) conference on Implementing Finch. The timetable for finalising the details of this complex policy is thus extremely short and does not allow for adequate discussion of its serious consequences. Despite this, academics can still play an important role in resisting the threats posed.

So, What is Happening?

In summary, academic journals are being moved from a ‘pay-to-read’ model to a ‘pay-to-say’ model.

Continue reading

Open Access: News and Reflections from the ACSS Conference

Last week, the first big public event discussing the Open Access policy announced in July was held at the Royal Statistical Society by the Academy of Social Sciences. If you are interested, many of the presentations from the event are already available online, with more write-ups to follow, as well as a promised YouTube video of the entire event. The programme promised and delivered a good range of speakers, and not least Dame Janet Finch herself.  I went along for the first day, thinking that this might be an open space to learn about the issues and have discussions about the policy, involving a wide range of affected parties.

Janet Finch

I did learn a lot, although what I mainly learned was that no one was really prepared to take any real responsibility for a policy to which a lot of eminent and well-informed people had very serious objections. Finch insisted that she had to stick to a brief which did not involve ‘destabilising’ the publishing system. No one was there to answer from either BIS or RCUK, who both adopted the policy immediately upon the publication of the report in July. HEFCE, who have not formally announced a position yet, however indicated at the conference that they are very likely to adopt the RCUK model for REF2020. Continue reading

Death To Open Access! Long Live Open Access!

A few weeks ago at the Millennium conference, some of us got together to talk about open access and the political economy of knowledge (re)production in our little corner of academia (“us” being Colin Wight, David Mainwaring, Nivi Manchanda, Nathan Coombs, Meera and me). Over the remainder of this week, we’ll be posting those reflections here for your delectation because, some discussion notwithstanding, labourers in today’s university-factories need to get talking about these things, and fast.

Open Access appears to be here. The Finch Report has recommended it, the Government has endorsed it, and there even seem to be some monies newly available for it. The battle is won, and the age of unfettered academic-public intercourse is upon us. Well, not quite. Finch’s preference for Gold Open Access, in which journals continue to receive revenues and make profits and in which academics (or their institutions) pay a fee of several thousand pounds per article for the pleasure (the so called Article Processing Charge (APC) system, which will receive greater attention in later posts), is deeply problematic (well-reasoned explanations for why available from Stevan Harnad and Peter Coles (Telescoper), with more qualified views, even cautious support, from Stephen Curry and Repository Man). Also, the monies aren’t new, and have instead been extracted from existing research budgets (and what a complete and utter surprise that is).

This is all cause for serious concern, and relates closely to the kinds of arguments that are developed and deployed in favour of Open Access (or, to be more provincial about it, Open IR). There are three kinds of arguments for opening up the journal system, arguments from access, ethics and cost, and we are in danger of letting the first overwhelm the different, and better, cases made in the name of the second and third.

The first set of arguments has to do with the problem of access: that the journal system is broken because it creates barriers to the circulation of academic knowledge. The journal as usually conceived was an ingenious and appropriate method for collating and distributing knowledge in the 18th century, but is now redundant. And yet journals – with their pay walls – remain the only (or at least the principal) vector for success or employment in academic life. This is doubly problematic since the various metrics that journal comparisons and prestige enable themselves then become vectors of discipline and control, even in the face of the many, many reasoned objections to such measures.

Continue reading

Academia in the Age of Digital Reproduction; Or, the Journal System, Redeemed

It took at least 200 years for the novel to emerge as an expressive form after the invention of the printing press.

So said Bob Stein in an interesting roundtable on the digital university from back in April 2010. His point being that the radical transformations in human knowledge and communication practices wrought by the internet remain in their infancy. Our learning curves may be steeper but we haven’t yet begun to grapple with what the collapsing of old forms of social space means. We tweak and vary the models that we’re used to, but are generally cloistered in the paradigms of print.

When it comes to the university, and to the journal system, this has a particular resonance. Academics find themselves in a strange and contradictory position. They are highly valued for their research outputs in the sense that this is what determines their reputation and secures their jobs (although this is increasingly the value of the faux-market and the half-assed quality metric). This academic authority, won by publications, is also, to some extent, what makes students want to work with them and what makes them attractive as experts for government, media and civil society. They are also highly valued in a straight-forward economic sense by private publishing houses, who generate profit from the ability to sell on the product of their labour (books and articles) at virtually no direct remuneration, either for the authors or for those peer reviewers who guarantee a work’s intellectual quality. And yet all (OK, most) also agree that virtually nobody reads this work and that peer review is hugely time-consuming, despite being very complicated in its effects. When conjoined with the mass noise of information overload and the extension and commercialisation of higher education over the last decades, our practices of research, dissemination and quality control begin to take on a ludicrous hue. As Clifford Lynch nicely puts it, “peer review is becoming a bottomless pit for human effort”.

This is an attempt to explore in more detail what the potentialities and limits are for academic journals in the age of digital reproduction. Once we bracket out the sedimented control of current publishers, and think of the liveliness of intellectual exchange encountered through blogs and other social media, a certain hope bubbles up. Why not see opportunity here? Perhaps the time is indeed ripe for the rebirth of the university press, as Martin Weller argues:

the almost wholesale shift to online journals has now seen a realignment with university skills and functions. We do run websites and universities are the places people look to for information (or better, they do it through syndicated repositories). The experience the higher education sector has built up through OER, software development and website maintenance, now aligns nicely with the skills we’ve always had of editing, reviewing, writing and managing journals. Universities are the ideal place now for journals to reside.

Continue reading

Commodification, the Academic Journal Racket and the Digital Commons

David, my erstwhile ‘parasitic overlord’ from when I was co-editing Millennium, points me to some posts by Kent Anderson of the Society for Scholarly Publishing, who defends the industry on a number of grounds from Monbiot’s polemic against the journal racket. The comments threads on both pieces are populated by academics who agree with Monbiot and by publishing industry colleagues who agree with Anderson (and who alternate between dismissing and being personally offended by the original Monbiot column). The core counter-argument is that this anti-corporate, out of touch, ‘wannabe-academic’ day dreaming is old hat, and stands up no better now than it did when it was demolished at some unspecified point in the past.

Most crucially, Monbiot’s central exhibit (that companies consistently make 30-40% margins on the distribution of work already paid for by the public purse) is almost entirely passed over. Anderson coyly suggests that maybe publisher margins are that high, but maybe they’re not. Despite working rather closer to the heart of matters financial than do the rest of us, he provides no settling of accounts either way. In any case, however much it costs, and however much publishers make, it’s good value, apparently by definitional fiat. Since libraries keep paying the money, and since academics keep submitting papers, it is ‘idealism’ (remember that?) to complain about the current balance of power.

This is the familiar circular logic of neoliberal reason: privatised arrangements are beneficial because they will make the system more efficient and less costly. But if the rate of profit does not fall in line with the expectations of open competition, then it must mean that the rates charged are true equilibrium prices. Nevertheless, complainants citing high margins are referred to the benefits of privatised arrangements and assured that competition will bring prices down. Even though £200 million each year, or 10% of all research funding distributed by HEFCE (the academic funding council for England), ends up being spent on journal and database access by academic libraries. [1]

Continue reading