Metrics: An Addendum on RAE / REF

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading

HEFCE, the REF, Open Access and Journals in Politics, IR and Political Theory

UPDATE (8 September 2014): Lee originally wrote this as a guest post – providing some much needed concrete detail on journal open access policies – but is now with us for good.


Following the launch of HEFCE’s consultation on Open Access for any post-2014 REF, and the generally positive reaction to it here, I examined the potential implications of HEFCE’s proposals for journal publishing in Politics, IR and Political Theory. I wanted to know whether the serious threat in HEFCE’s earlier proposals – notably the rush for ‘gold’ OA and associated Article Processing Charges (APCs) – had been eliminated by a downgrading of the proposals to permit ‘green’ OA, by depositing a ‘final accepted version’ (FAV, i.e. post-peer review, but pre-type setting) into an institutional repository. I also wanted to see what embargoes journals placed on FAVs (i.e. how long after the ‘Version of Record’ (VoR) is published in the journal the FAV can be made public); what re-use was permitted (what sort of licensing); and also to compare this route with the ‘gold’ access favoured by the Finch Report and the RCUK policy. I also wanted to gather information that could be used as part of a ‘soft boycott’ of OA-resistant outlets.

To do this I selected 57 journals that broadly represent the ‘top’ journals in the three subfields. I used composite rankings from Google Scholar, the ISI Citations Index, and surveys of scholars where available, and got feedback on an initial draft. The list isn’t intended to be definitive, just to give us a better sense of where the ‘big journals’ in which many scholars aspire to publish actually stand on OA.

It is not easy to get this information. Policies can vary by journal, not merely by publisher, and their websites are often opaque on the issue of self-archiving, particularly in terms of licensing. This may change if, as seems likely, HEFCE forges ahead on OA; but publishers also need to be pushed to display clearer information. The exaggerated nature of the Finch Report’s estimate of UK HE’s market power to change publishing models is underscored by the fact that US journals tend not to provide the information I was looking for (my thanks to Sarah Molloy for help with this).

The results are presented below for Politics, IR and Political Theory (click to enlarge each image). There are a lot of complexities with various journals which are shown in the full spreadsheet of results which you can see here; the spreadsheet also lets you reorder the information by different criteria.

OA - IR

Journal Open Access Policies for International Relations.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Politics.

Journal Open Access Policies for Political Theory.

Journal Open Access Policies for Political Theory.

Several conclusions can be drawn.

(1) Most importantly, Pablo and Meera were basically right: the threat of HEFCE rushing to a ‘pay to say’ approach with hugely detrimental financial consequences for universities and the potential for internal rationing of APCs, has been defeated, for now (although RCUK’s policies remain intact). Some journals, particularly in the US, don’t seem to support green FAV self-archiving, or have gold APC routes – this is a problem because they would be non-compliant with HEFCE’s proposals; so we will need to push for exemptions on this score. However, by and large, where they allow FAV self-archiving, none of the journals appear to charge a fee; HEFCE’s current proposals now intend to compel more researchers to do what they are already entitled to do. So long as this does not change, HEFCE’s proposals should not involve ruinous costs or lead to APC rationing.

Continue reading