Nu-Bureaucracy

Can Metrics Be Used Responsibly? Why Structural Conditions Push Against This

/ Meera / 11 Comments
Not waving, exactly, but...

Not waving, exactly, but…

Today, the long-awaited ‘Metric Tide’ report from the Independent Review of the Role of Metrics in Research Assessment and Management was published, along with appendices detailing its literature review and correlation studies. The main take-away is: IF you’re going to use metrics, you should use them responsibly (NB NOT: You should use metrics and use them responsibly). The findings and ethos are covered in the Times Higher and summarised in Nature by the Review Chair James Wilsdon, and further comments from Stephen Curry (Review team) and Steven Hill (HEFCE) are published. I highly recommend this response to the findings by David Colquhoun. You can also follow #HEFCEMetrics on Twitter for more snippets of the day. Comments by Cambridge lab head Professor Ottoline Leyser were a particular highlight.

I was asked to give a response to the report at today’s launch event, following up on the significance of mine and Pablo’s widely endorsed submission to the review. I am told that the event was recorded by video and audio so I will add links to that when they show up. But before then, a short summary record of the main points I made: Continue reading →

Metrics: An Addendum on RAE / REF

/ Meera / 6 Comments

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading →

Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

/ Meera / 178 Comments

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to metrics@hefce.ac.uk). If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at]soas.ac.uk or p.c.kirby[at]sussex.ac.uk before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.

Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex

Summary

Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading →

Getting Somewhere: HEFCE Proposals on Open Access for a Post-2014 Research Excellence Framework

/ Meera / 15 Comments

zimbabwe-press

This week, the UK’s Higher Education Funding Council for England (HEFCE) published their formal proposals for including an open access requirement in any post-2014 Research Excellence Framework (REF). Responses to this will be accepted until 30th October 2013. These proposals follow a pre-consultation letter and set of responses which were submitted earlier in the year (link to University of Cambridge response).

Following up our concerns about the policy raised over the last few months (here and here, further posts here) the present iteration represents a decent outcome on some of the details, not least because it defers quite a few of them. That these issues have been deferred does not mean that they do not matter; rather it means that the battles on them will be fought elsewhere – with universities, with journal boards, with learned societies, with publishers and their lawyers and so on. Moreover, there is no cause for complacency around the broader political economy of scholarly publishing, which remains wasteful, restrictive and inequitable on many fronts. And of course, the pernicious REF exercise itself, which this government signalled it would review, must be itself vigorously contested (more on this to come).

The Requirements

The proposals are to require that any REF-submitted journal article or conference proceeding published after 2016 must be made available in the final post-peer reviewed version from an institutional repository at the point of acceptance (or publication). This in line with the previous agenda of RCUK and others, and maintains journal exclusivity by accepting substantial embargo times on truly open (read: public) viewing of these deposited versions. So a paper “immediately” placed in an institutional repository may still not be viewable for up to 24 months in the Arts, Humanities and Social Sciences (recall that this length of permissible embargo was extended in the face of publisher lobbying), although it may be possible to request papers directly from the author on an individual basis. This version of mandated open access is applicable only if the address line of the author is a UK HEI at the time of publication, and cases for exceptions can be made. HEFCE’s assessment that this should be broadly achievable under current provisions is reasonable, and represents a good path for opening up access to research under present conditions. From the perspective of maximising green OA, Stevan Harnad’s response is as ever highly incisive. Below are some reflections on the present state of play.

What Has Been Won and Deferred

First, the old green/gold battle is now (nearly) over, and with it the concerns that academics would routinely pay exorbitant author fees to have their research published. Continue reading →

Open Access: HEFCE, REF2020 and the Threat to Academic Freedom

/ Pablo K / 67 Comments

This is the text of a document prepared by Meera and me on Article Processing Charges as currently understood and the serious risks we think they pose to academic freedom and funding, broadly understood (previous discussed by several contributors to our open access series). It is also available as a pdf, and we encourage academics to think carefully about the issues foregrounded, and to act accordingly.

Applegarth Press

Summary

  • The Government is pushing academic publishing to a ‘pay-to-say’ model in order to achieve open access to publicly funded research
  • This ‘gold’ route to open access, which levies Article Processing Charges (as proposed in the Finch Report and taken up by RCUK and HEFCE) poses a major problem for academics in the UK:
    • It threatens academic freedom through pressures on institutions to distribute scarce APC resources and to judge work by standards other than peer review
    • It threatens research funding by diverting existing funds into paying for publications (and private journal profits) rather than into research
    • It increases academic inequality both across and within institutions, by linking prestige in research and publishing to the capacity to pay APCs, rather than to academic qualities
    • It threatens academic control of research outputs by allowing for commercial uses without author consent
  • In response, academics should:
    • Practice and lobby for ‘green’ open access of all post-peer reviewed work within journals and institutions
    • Lobby against proposed restrictions on REF2020 and against compliance pressure for ‘gold’ open access
    • Demand clear policies from Universities around open access funds
    • Ensure institutional resources are not unnecessarily spent on APCs
    • Protect the integrity of scholarly journals by rejecting the pressure for ‘pay-to-say’ publishing

Open Access: Rushing Implementation

Many academics have been ardent supporters of the open access principle (that peer-reviewed academic work should be freely available and easily accessible to anyone), and were excited when the Government made steps to advance it. However, it has become clear that the implementation of this policy via REF2020 will have very serious negative consequences for all academic authors and institutions, unless authors and institutions themselves start to take action and make their voices heard. It is critical that academics understand what is happening and lobby our pro-VCs of Research, our VCs and Universities UK to defend both academic freedom and open access.

The timescale for action and decision-making is now incredibly short. Several policies, including that of the Government and of RCUK were declared immediately with the release of the Finch Report, totally accepting its views without wider consultation. HEFCE is going to open and close a very quick consultation period early in 2013 in order to issue guidance ahead of REF2020. Some universities have been given until March 2013 to determine what to do with open access funds that they were given in November. And it was only on 29 November 2012 that the first indications from HEFCE were given as to their intentions, at the Academy of Social Sciences (ACSS) conference on Implementing Finch. The timetable for finalising the details of this complex policy is thus extremely short and does not allow for adequate discussion of its serious consequences. Despite this, academics can still play an important role in resisting the threats posed.

So, What is Happening?

In summary, academic journals are being moved from a ‘pay-to-read’ model to a ‘pay-to-say’ model.

Continue reading →

Open Access: News and Reflections from the ACSS Conference

/ Meera / 31 Comments

Last week, the first big public event discussing the Open Access policy announced in July was held at the Royal Statistical Society by the Academy of Social Sciences. If you are interested, many of the presentations from the event are already available online, with more write-ups to follow, as well as a promised YouTube video of the entire event. The programme promised and delivered a good range of speakers, and not least Dame Janet Finch herself.  I went along for the first day, thinking that this might be an open space to learn about the issues and have discussions about the policy, involving a wide range of affected parties.

Janet Finch

I did learn a lot, although what I mainly learned was that no one was really prepared to take any real responsibility for a policy to which a lot of eminent and well-informed people had very serious objections. Finch insisted that she had to stick to a brief which did not involve ‘destabilising’ the publishing system. No one was there to answer from either BIS or RCUK, who both adopted the policy immediately upon the publication of the report in July. HEFCE, who have not formally announced a position yet, however indicated at the conference that they are very likely to adopt the RCUK model for REF2020. Continue reading →

One Size Fits All?: Social Science and Open Access

/ Guest Authors / 18 Comments

The third post in our small series on open access, publication shifts on the horizon and how it all matters to IR and social science, this time by David Mainwaring  (Pablo’s post was first, then Colin Wight’s, and following David came Nivi Manchanda, Nathan Coombs and our own Meera). David is a Senior Editor at SAGE with responsibility for journals in politics and international studies. So he oversees journals like Millennium and European Journal of International Relations amongst others (which is how we know him), and can thus offer a close reading of movements within the academic publishing industry. Images by Pablo.

Open Access is the talk of the academic town. The removal of barriers to the online access and re-use of scholarly research is being driven by a cluster of technological, financial, moral and commercial imperatives, and the message from governments and funding agencies is clear: the future is open. What is much less clear is exactly what sort of open future social scientists would benefit from, let alone what steps need to be taken in order to transition away from the existing arrangements of scholarly communication and validation. Here the conversation is in its relative infancy, characterised at this point by a great deal of curiosity, anticipation, confusion, and the shock of the new. What it needs to move towards is a recognition and coordinated response to the fact that although social science may share the same open access goal as the STEM disciplines, the motivations for travelling down that path are not identical, and the context – especially in terms of research funding – is significantly different. The roundtable discussion at the Millennium conference at the LSE on 20th October was an attempt to explore these issues specifically from an IR perspective; further such events (such as those being run by the AcSS and LSE this autumn) are to be warmly welcomed as a means of building broader understanding of the issues among social scientists and facilitating strategic thinking.

Social Science and the Open Access Debate

The long-running debate about how scholarly research communication should be funded and transmitted has been, and remains, a discussion conducted primarily by those working in STEM. Most of the blogosphere’s best-known voices on open access, the likes of Mike Taylor, Michael Eisen, Peter Murray-Rust, Björn Brembs, Cameron Neylon, Kent Anderson, Stephen Curry and Tim Gowers, have backgrounds in STEM research or publishing. (Notable exceptions are the philosopher Peter Suber, who heads the Harvard Open Access Project, and self-archiving advocate Stevan Harnad, a cognitive scientist). To dip into the often heated debates on open access can leave you with the strong impression that, despite the occasional nod to social science and the humanities, the frame of reference is proper, rigorous, natural scientific research, the kind carried out in a laboratory that leads to medical advances and the development of new technologies.

By and large social scientists – and arts and humanities scholars, to whom many of the points raised in this piece apply equally – have had a back seat in this conversation, and the development of open access awareness and capabilities has been slow. Many leading social science journals continue to be distributed in print form due to subscriber demand well over a decade after the launch of their online editions. The American Political Science Association’s 2009 book Publishing Political Science devotes just two out of over 250 pages to open access, and fewer than 10% of the nearly 13,000 signatories of the ‘Cost of Knowledge‘ boycott of Elsevier were social scientists, despite the company’s position as one of the world’s leading publishers of social science journals. That’s not to say that there is nothing going on: the Social Science Research Network has acted as a site for open paper sharing since 1994; there are active ‘open’ movements in disciplines including IR and economics; and the Directory of Open Access Journals lists more than 1,600 social science titles. To date, however, very few of the latter have been able to break into the higher echelons of profile or reputation within their fields.

Social Science and Open Access Mandates

Over the last eighteen months a series of events, including George Monbiot’s polemic in The Guardian and the defeat in the US of the Research Works Act, stirred for the first time a significant consciousness among social scientists about open access. This awareness increased dramatically – in the UK at least – with the publication in June 2012 of the final report of a committee set up by the Government to examine access to published research findings. Continue reading →

What We Talked About At ISA: The U.S. Military-Industrial Complex, Part 2

/ Srdjan / 7 Comments

Part two of a post on my presentation at this year’s ISA. Part one is here.

So what would be the normative-political case for the Military-Industrial Complex (MIC)? As Ledbetter notes, the defence industry never had a shortage of defenders, proponents, beneficiaries, and apologists. Various critiques of the MIC notwithstanding, numerous American commentators are now firmly united in the belief that their country needs a large defence budget in order to protect and project its identities and interests in the world. According to Maddow, this collective belief had a lot to do with the discursive and institutional success of the infamous “Team B” reports on Soviet power, which so profoundly enthused Ronald Reagan and his administration, leading to the gigantic military buildup in the 1980s. Maddow’s assessment is worth citing at length:

The Think Tanks and Very Important Committees of the permanent national security peanut gallery are now so mature and entrenched that almost no one thinks they’re creepy anymore, and national security liberals have simply decided it’s best to add their own voices to them rather than criticize them. But like we lefties learned in trying (and failing) to add a liberal network to the all-right-wing, decades-old medium of political talk radio, the permanent defense gadfly world can’t really grow a liberal wing. It’s an inherently hawkish enterprise. Where’s the inherent urgency in arguing that the threats aren’t as bad as the hype, that military power is being overused, that the defense budget could be safely and wisely scaled back, that maybe this next war doesn’t need us? The only audience for defense wonkery is defense enthusiasts, and they’re not paying the price of admission to hear that defense is overrated.

But knotted into the right-wing discourse on defence spending is a number of corollary arguments that are associated with a variety of lefty positions in the U.S. context.  America’s mainstream media outlets rarely fail to acknowledge how the twinning of the country’s economic and armed forces not only creates high-skilled jobs, but also – and critically – keeps them in the country. The move is mainly rhetorical. Not only have successive U.S. administrations encouraged American defence industry to globalize, but there is also little evidence to suggest that defence spending creates more jobs relative to spending on, say, health care or education (see, for example, Pollin and Garrett-Peltier, 2011). I would suggest, then, that what lies behind contemporary pro-MIC arguments is, in fact, a creative and complex combination of certain economic theories, (realist?) beliefs in war (or the threat of war) as a manifest destiny of the international system, as well as an overarching (liberal?) commitment to a powerful, sovereign state capable of exercising global leadership (aka., a “force for good”, in still favoured New Labour parlance.)

Let us revisit the pro-MIC rhetoric from the era of “Team B.” In a footnote, Ledbetter directs the reader to The Lonely Warriors (1970) by John Stanley Baumgartner, who is described as “one notable true defender of the MIC.” Written by an expert in public management and business administration, Baumgartner’s book makes three arguments for the MIC: 1) defending the free world is a moral thing to do (“Sputnik is only one example of the reasons for MIC”); 2) by definition, defence is a big enterprise and all big enterprises (directly or indirectly, the MIC employs one in ten Americans) occasionally make big mistakes, especially when they respond to the murky and changing specifications set by the government (“the tiger” or “the monster”) and its contracting officers; and last, 3) “unconscionable profit” is not so unconscionable in comparative terms (profit on sales, profit on investment, price/earning ratios etc. tend to be below the industrial average).

Continue reading →

What We Talked About At ISA: From #occupyirtheory to #OpenIR?

/ Pablo K / 5 Comments

A write up of my comments at the #occupyirtheory event in San Diego. The event itself was both hope-filled and occasionally frustrating, not least for the small group of walk-outs, apparently ‘political’ ‘scientists’ lacking in any conception of what it actually means to engage in the political (note: this bothered me especially, but was a rather minor irritation in the grander scheme of things). Despite the late hour, there were between 40 and 60 people there throughout, and a number of very positive things have come of it. It looks like there’ll be some gathering at BISA/ISA to discuss further, and we’re pitching something for the Millennium conference on some of the themes addressed below, and there will of course be ISA 2013 too. In the meantime, there’s the Facebook group, the blog, and a mailing list. The term OpenIR is owed to Kathryn Fisher, and seems to several of us to be a better umbrella term for the many things we want to address in the discipline and the academy. I also just want to give a public shout-out to Nick, Wanda, Robbie and Meera for doing so much on this.

The #occupy practice/meme has antecedents. Physical manifestations of a ‘public’, horizontalism, prefigurative politics and more can be traced in all sorts of histories. One such lineage is the foreshadowing of Zucotti Park in recent struggles over education. Take the slogan in March 2010 over privatisation at the University of California, which was ‘STRIKE / OCCUPY / TAKEOVER’. Or Middlesex, where students resisting the dismantling of the Philosophy Department in that same year unfurled a banner during their occupation, one that proclaimed: ‘THE UNIVERSITY IS A FACTORY! STRIKE! OCCUPY!’.

I want briefly, then, to think about the space of the university in our discussions of #occupy. There have been rich and suggestive calls to re-politicise ourselves as academic-activists, to look again at our work and its claims, and to turn our abilities, such as they are, to projects of resistance and transformation. But we risk a displacement. When we talk of ‘the street’, or politics enacted in the reconfigured space of #occupy, or of the ‘real world’ that we must be relevant to, we already miss the university itself as that factory in which we labour. We are tempted by a view of ourselves as leaving ivory towers to do politics, instead of seeing those towers themselves as spaces of politics. As if our institutions and practices were not already part of the world.

Whether you see #occupy as transformational or nor, or whether you simply prefer a different vocabulary, I think a demand remains: a demand to politicise our own positionality. This politicisation can have many dimensions, but I want to suggestively highlight four, each being a sphere in which we should be diagnosing and transforming our own practices.

Continue reading →

The AHRC Writes Back (Kind Of)

/ Pablo K / 1 Comment

Today the Arts & Humanities Research Council responded to yesterday’s piece in The Observer claiming that the government had readjusted the rules (specifically The Haldane Principle) to increase their control over the direction of research in the UK within the state’s ‘national priorities’. Shorter version: the Tories didn’t pressure us, we’re completely independent and the funding to ‘The Big Society’ is coincedental.

Iain Pears is on the case. As he notes, it’s simply not convincing that the approved language suddenly appeared in the relevant documents unconnected with the political agenda of the governing party. The pressure may have been implicit, and the relationship might have been more informal and complicit than hierarchical, but the consequences for research are much the same.

But the AHRC not only wants to defend itself from these specific charges but also to maintain the legitimacy of the government setting overall priorities. Once again, the exact mechanics are expected to be taken on trust. Indeed, in 2009 a Commons Select Committee (Innovation,  Universities, Science and Skills, since you asked) addressed this exact point. While agreeing ‘in theory’ that the government had a role in setting overarching strategy, the relevant MPs (hardly a selection of Parliamentary rebels) put their collective figure on the aspects of policy that still concern us most:

Continue reading →