Can Metrics Be Used Responsibly? Why Structural Conditions Push Against This

Not waving, exactly, but...

Not waving, exactly, but…

Today, the long-awaited ‘Metric Tide’ report from the Independent Review of the Role of Metrics in Research Assessment and Management was published, along with appendices detailing its literature review and correlation studies. The main take-away is: IF you’re going to use metrics, you should use them responsibly (NB NOT: You should use metrics and use them responsibly). The findings and ethos are covered in the Times Higher and summarised in Nature by the Review Chair James Wilsdon, and further comments from Stephen Curry (Review team) and Steven Hill (HEFCE) are published. I highly recommend this response to the findings by David Colquhoun. You can also follow #HEFCEMetrics on Twitter for more snippets of the day. Comments by Cambridge lab head Professor Ottoline Leyser were a particular highlight.

I was asked to give a response to the report at today’s launch event, following up on the significance of mine and Pablo’s widely endorsed submission to the review. I am told that the event was recorded by video and audio so I will add links to that when they show up. But before then, a short summary record of the main points I made: Continue reading

Advertisements

Metrics: An Addendum on RAE / REF

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading

Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to metrics@hefce.ac.uk). If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at]soas.ac.uk or p.c.kirby[at]sussex.ac.uk before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.


Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex

Summary

Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading

HEFCE, the REF, Open Access and Journals in Politics, IR and Political Theory

UPDATE (8 September 2014): Lee originally wrote this as a guest post – providing some much needed concrete detail on journal open access policies – but is now with us for good.


Following the launch of HEFCE’s consultation on Open Access for any post-2014 REF, and the generally positive reaction to it here, I examined the potential implications of HEFCE’s proposals for journal publishing in Politics, IR and Political Theory. I wanted to know whether the serious threat in HEFCE’s earlier proposals – notably the rush for ‘gold’ OA and associated Article Processing Charges (APCs) – had been eliminated by a downgrading of the proposals to permit ‘green’ OA, by depositing a ‘final accepted version’ (FAV, i.e. post-peer review, but pre-type setting) into an institutional repository. I also wanted to see what embargoes journals placed on FAVs (i.e. how long after the ‘Version of Record’ (VoR) is published in the journal the FAV can be made public); what re-use was permitted (what sort of licensing); and also to compare this route with the ‘gold’ access favoured by the Finch Report and the RCUK policy. I also wanted to gather information that could be used as part of a ‘soft boycott’ of OA-resistant outlets.

To do this I selected 57 journals that broadly represent the ‘top’ journals in the three subfields. I used composite rankings from Google Scholar, the ISI Citations Index, and surveys of scholars where available, and got feedback on an initial draft. The list isn’t intended to be definitive, just to give us a better sense of where the ‘big journals’ in which many scholars aspire to publish actually stand on OA.

It is not easy to get this information. Policies can vary by journal, not merely by publisher, and their websites are often opaque on the issue of self-archiving, particularly in terms of licensing. This may change if, as seems likely, HEFCE forges ahead on OA; but publishers also need to be pushed to display clearer information. The exaggerated nature of the Finch Report’s estimate of UK HE’s market power to change publishing models is underscored by the fact that US journals tend not to provide the information I was looking for (my thanks to Sarah Molloy for help with this).

The results are presented below for Politics, IR and Political Theory (click to enlarge each image). There are a lot of complexities with various journals which are shown in the full spreadsheet of results which you can see here; the spreadsheet also lets you reorder the information by different criteria.

OA - IR

Journal Open Access Policies for International Relations.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Political Theory.

Journal Open Access Policies for Political Theory.

Several conclusions can be drawn.

(1) Most importantly, Pablo and Meera were basically right: the threat of HEFCE rushing to a ‘pay to say’ approach with hugely detrimental financial consequences for universities and the potential for internal rationing of APCs, has been defeated, for now (although RCUK’s policies remain intact). Some journals, particularly in the US, don’t seem to support green FAV self-archiving, or have gold APC routes – this is a problem because they would be non-compliant with HEFCE’s proposals; so we will need to push for exemptions on this score. However, by and large, where they allow FAV self-archiving, none of the journals appear to charge a fee; HEFCE’s current proposals now intend to compel more researchers to do what they are already entitled to do. So long as this does not change, HEFCE’s proposals should not involve ruinous costs or lead to APC rationing.

Continue reading

Getting Somewhere: HEFCE Proposals on Open Access for a Post-2014 Research Excellence Framework

zimbabwe-press

This week, the UK’s Higher Education Funding Council for England (HEFCE) published their formal proposals for including an open access requirement in any post-2014 Research Excellence Framework (REF). Responses to this will be accepted until 30th October 2013. These proposals follow a pre-consultation letter and set of responses which were submitted earlier in the year (link to University of Cambridge response).

Following up our concerns about the policy raised over the last few months (here and here, further posts here) the present iteration represents a decent outcome on some of the details, not least because it defers quite a few of them. That these issues have been deferred does not mean that they do not matter; rather it means that the battles on them will be fought elsewhere – with universities, with journal boards, with learned societies, with publishers and their lawyers and so on. Moreover, there is no cause for complacency around the broader political economy of scholarly publishing, which remains wasteful, restrictive and inequitable on many fronts. And of course, the pernicious REF exercise itself, which this government signalled it would review, must be itself vigorously contested (more on this to come).

The Requirements

The proposals are to require that any REF-submitted journal article or conference proceeding published after 2016 must be made available in the final post-peer reviewed version from an institutional repository at the point of acceptance (or publication). This in line with the previous agenda of RCUK and others, and maintains journal exclusivity by accepting substantial embargo times on truly open (read: public) viewing of these deposited versions. So a paper “immediately” placed in an institutional repository may still not be viewable for up to 24 months in the Arts, Humanities and Social Sciences (recall that this length of permissible embargo was extended in the face of publisher lobbying), although it may be possible to request papers directly from the author on an individual basis. This version of mandated open access is applicable only if the address line of the author is a UK HEI at the time of publication, and cases for exceptions can be made. HEFCE’s assessment that this should be broadly achievable under current provisions is reasonable, and represents a good path for opening up access to research under present conditions. From the perspective of maximising green OA, Stevan Harnad’s response is as ever highly incisive. Below are some reflections on the present state of play.

What Has Been Won and Deferred

First, the old green/gold battle is now (nearly) over, and with it the concerns that academics would routinely pay exorbitant author fees to have their research published. Continue reading

Open Access: HEFCE, REF2020 and the Threat to Academic Freedom

This is the text of a document prepared by Meera and me on Article Processing Charges as currently understood and the serious risks we think they pose to academic freedom and funding, broadly understood (previous discussed by several contributors to our open access series). It is also available as a pdf, and we encourage academics to think carefully about the issues foregrounded, and to act accordingly.


Applegarth Press

Summary

  • The Government is pushing academic publishing to a ‘pay-to-say’ model in order to achieve open access to publicly funded research
  • This ‘gold’ route to open access, which levies Article Processing Charges (as proposed in the Finch Report and taken up by RCUK and HEFCE) poses a major problem for academics in the UK:
    • It threatens academic freedom through pressures on institutions to distribute scarce APC resources and to judge work by standards other than peer review
    • It threatens research funding by diverting existing funds into paying for publications (and private journal profits) rather than into research
    • It increases academic inequality both across and within institutions, by linking prestige in research and publishing to the capacity to pay APCs, rather than to academic qualities
    • It threatens academic control of research outputs by allowing for commercial uses without author consent
  • In response, academics should:
    • Practice and lobby for ‘green’ open access of all post-peer reviewed work within journals and institutions
    • Lobby against proposed restrictions on REF2020 and against compliance pressure for ‘gold’ open access
    • Demand clear policies from Universities around open access funds
    • Ensure institutional resources are not unnecessarily spent on APCs
    • Protect the integrity of scholarly journals by rejecting the pressure for ‘pay-to-say’ publishing

Open Access: Rushing Implementation

Many academics have been ardent supporters of the open access principle (that peer-reviewed academic work should be freely available and easily accessible to anyone), and were excited when the Government made steps to advance it. However, it has become clear that the implementation of this policy via REF2020 will have very serious negative consequences for all academic authors and institutions, unless authors and institutions themselves start to take action and make their voices heard. It is critical that academics understand what is happening and lobby our pro-VCs of Research, our VCs and Universities UK to defend both academic freedom and open access.

The timescale for action and decision-making is now incredibly short. Several policies, including that of the Government and of RCUK were declared immediately with the release of the Finch Report, totally accepting its views without wider consultation. HEFCE is going to open and close a very quick consultation period early in 2013 in order to issue guidance ahead of REF2020. Some universities have been given until March 2013 to determine what to do with open access funds that they were given in November. And it was only on 29 November 2012 that the first indications from HEFCE were given as to their intentions, at the Academy of Social Sciences (ACSS) conference on Implementing Finch. The timetable for finalising the details of this complex policy is thus extremely short and does not allow for adequate discussion of its serious consequences. Despite this, academics can still play an important role in resisting the threats posed.

So, What is Happening?

In summary, academic journals are being moved from a ‘pay-to-read’ model to a ‘pay-to-say’ model.

Continue reading

Open Access: News and Reflections from the ACSS Conference

Last week, the first big public event discussing the Open Access policy announced in July was held at the Royal Statistical Society by the Academy of Social Sciences. If you are interested, many of the presentations from the event are already available online, with more write-ups to follow, as well as a promised YouTube video of the entire event. The programme promised and delivered a good range of speakers, and not least Dame Janet Finch herself.  I went along for the first day, thinking that this might be an open space to learn about the issues and have discussions about the policy, involving a wide range of affected parties.

Janet Finch

I did learn a lot, although what I mainly learned was that no one was really prepared to take any real responsibility for a policy to which a lot of eminent and well-informed people had very serious objections. Finch insisted that she had to stick to a brief which did not involve ‘destabilising’ the publishing system. No one was there to answer from either BIS or RCUK, who both adopted the policy immediately upon the publication of the report in July. HEFCE, who have not formally announced a position yet, however indicated at the conference that they are very likely to adopt the RCUK model for REF2020. Continue reading