Metrics: An Addendum on RAE / REF

24 Jun

Not everything that counts can be counted, and not everything that can be counted counts...

We have had overwhelming support from a wide range of academics for our paper on why metrics are inappropriate for assessing research quality (200+ as of June 22nd). However, some have also posed interesting follow-up questions on the blog and by email which are worth addressing in more depth. These are more REF-specific on the whole and relate to the relationship between the flaws in the current system and the flaws in the proposed system. In my view the latter still greatly outweigh the former but it is useful to reflect on them both.

Current REF assessment processes are unaccountable and subjective; aren’t metrics a more transparent, public and objective way of assessing research?

The current REF involves, as the poser of the question pointed out, small groups of people deliberating behind closed doors and destroying all evidence of their deliberations. The point about the non-transparency and unaccountability of this process is an important one to keep in mind.

The question is then posed, are metrics more transparent, public and objective? On a surface level, metrics are more ‘transparent’ because they are literally visible (public) and given a number, making them easily rankable. But what they represent, as we argued in our paper, is fundamentally non-transparent given the wide variety of reasons there might be for citing work, and more besides those we cited. In fact, it is the very simulation of transparency in the use of a numerical marker that becomes threatening to the act of actually reading work for assessment purposes. Continue reading

Acting Time; Or, Ending Sexual Violence in Conflict

17 Jun

Pitt Jolie ESVC PicturesThe attention lavished on sexual violence in conflict last week was in many ways unprecedented. As well as convening the largest ever gathering of officials, NGOs and other experts for the Global Summit on Ending Sexual Violence in Conflict, co-chairs William Hague (Foreign Secretary of the United Kingdom of Great Britain and Northern Ireland) and Angelina Jolie (Special Envoy of the United Nations High Commissioner on Refugees) also generated very many pages – both print and digital – of commentary. In some myopic quarters, that achievement was in itself a distraction from the really important politics of blossoming conflict in Iraq. Such views should remind us that there are still those who insist on seeing gender violence as marginal to international peace and security. Worthy, yes, “no doubt important”, obviously a cause for concern, and so on, but naturally not the real deal.

Since the Summit’s close on Friday, there have also been criticisms of a different sort. A protest on the first day drew attention to the asylum and refugee policies of Her Majesty’s Government, and the ways in which survivors of sexual violence were being mistreated on the British mainland. The Foreign Office raised awareness in part through one-dimensional stories of crazy monsters in the hinterlands of barbarism. The “weapon of war” framework was ubiquitous, but no less problematic for that (see also). Although the Summit made space for youth delegates, UN entities, amateur hackers, foreign ministers, survivors, doctors, lawyers, celebrities, military officers and the odd NGO, academics (and our directly relevant research) were barely at the table. Some myths were therefore recycled. Delegates insisted on using rape survivors as props for their own journeys of self-discovery. I met a women in Panzi Hospital and what she told me broke my heart, etcetera. Some national representatives seemed only just to have discovered the existence of United Nations Security Council Resolution 1325, which urged the participation of women in military and political settings at all levels. That was, um, 14 years ago. John Kerry, amongst others, appeared to believe that rape in war was not yet illegal, but that we could make it so if we really put our minds to it.

The Fringe events were themselves a source of considerable disappointment. Angelina opened proceedings by assuring us that “our” institutions protected us from rape, and prosecuted it ably when it did occur, whilst “they” (we all know who) need our help because they are confined to refugee camps. There was a staged ‘trial’ of the afore-mentioned Resolution 1325, in which an all-white panel of lawyers and faux-judges, including Cherie Booth QC, took the testimony of African witnesses. You could buy various goods made by (or meant to help) rape survivors in the “bustling” Fringe marketplace, and the official programme recommended that you “treat yourself” by doing so. All of this (including the less appalling and more considered exhibits) seemed removed from the set piece debates upstairs. If the Foreign Secretary really did refuse to meet with four Nobel Laureates - some of whom are themselves survivors of political rape – then clearly civil society (that vague but essential category) was being neglected.

Those accumulated complaints can be dismissed as relatively trivial if the Summit gets even some way to achieving its stated aim of ending sexual violence in conflict.[1]  Continue reading

Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

16 Jun

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to metrics@hefce.ac.uk). If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at]soas.ac.uk or p.c.kirby[at]sussex.ac.uk before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.


Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex

Summary

Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading

The Onuf Principles

30 May

Nicholas Onuf recently gave an interview over at e-IR. Several people of our acquaintance shared the tail-end of that, in which he is prompted to dispense career advice. Although opinions are indeed like assholes, these are good enough to elevate far above the gutter.


Original image by Stéphanie Saramago

Original image by Stéphanie Saramago

1. Preparing at length for classes does not make you a better teacher. Insofar as it dampens spontaneity, students will think you are boring; this will undercut the self-confidence you thought your lengthy preparations had purchased for you. And, of course, it steals valuable time from your scholarship.

2. Writing is a craft; writing well takes most of us a great deal of work. The usual practice is to think of a problem or issue, formulate a project, do ‘research,’ and then write it up. Bad idea. Keep writing at every stage, even if, in the end, you throw out most of what you have written. Writing makes the problem clearer, points up what more you need to do in the way of research, and, most of all, keeps your writing skills well-honed.

3. Don’t send sloppy, badly crafted papers out for review. As a frequent referee, I see them all too often. Many referees will punish you, not always consciously, for doing so, even if they think you are on to something. Once you think you have a well-crafted piece of work, do send it out, because most referees and editors take their duties seriously and will give you valuable feedback.

4. Be cautious about taking on collaborative projects. We all know that scholarship is a lonely occupation. Collaboration reduces the loneliness quotient and can result in better work than any of the collaborators could have produced on their own. It can also result in a piece of work that no one is entirely happy with. Sometimes collaboration causes damaging tension and bad feelings because of temperamental differences, greater or lesser commitment to the project, and perceived inequities in the distribution of work. All that said, collaborating with my brother on two book projects was hugely rewarding. That it might have been hugely risky never occurred to us.

5. Be even more cautious in participating in symposium projects. Their thematic foci may not match your interests very well; they tend to be superficially refereed and thus are not taken seriously; they also tend to disappear quickly from view. There are exceptions—symposia that mark major developments in the field—but you’ll have a pretty good idea if a particular symposium project has that potential. As a senior scholar, I contribute to symposia because it is fun to do projects with friends and I can afford the luxury. Most of all, avoid editing symposium volumes. This involves collaboration under the most difficult conditions. It is extraordinarily time consuming. Wrangling recalcitrant contributors is too often a thankless and disheartening responsibility.

6. Do not take on too many projects at one time. You will spread yourself too thin, miss deadlines, and make it all the more likely that you will succumb to the 90% rule—you run out of steam when any given project is 90% done and only needs some fine-tuning to be sent off. You will end up with a drawer full of nearly done projects that you have progressively lost interest in and will therefore never finish.

7. Dissertations are apprentice projects, immediately recognisable as such. Turning a dissertation into a book is probably the smart thing to do, but it will often take longer than writing the dissertation did. For most of us, it takes five years to write a good book; World of Our Making took me ten years. Whether you have that much time, institutionally speaking, is another matter.

8. Read every day. When I get up in the morning (early) and get my coffee, I read for 45 minutes. In my case, it has always been something that I do not have to read for whatever I am doing at the time. While this has broadened me immeasurably, for many scholars, a fixed time for reading is an opportunity — perhaps the only opportunity — to keep up on the literature in the field.

9. Whether to jump on a trend in the field’s scholarship, try anticipating a trend, come late to a trend but treat it critically, jump around from thing to thing, or plug away at something few others seem to be interested in is a tricky question, having much to do with temperament. It requires you to ask yourself how ambitious you are, how much you need validation from others, how long you can stayed focused on one thing, et cetera.

10. On the assumption that you are smarter than most people (or you would not be a scholar), seek out people whom you know to be smarter than you in various obvious ways. On the one hand, the more of these people you know, the less intimidating you will find them, and the more you will learn from them. On the other hand, knowing really smart people will remind you of your own limitations and help you be less arrogant. Arrogance is, of course, a constant hazard in our line of work.

Bad Infinity?: Hans J. Morgenthau’s Double Critique of Depoliticisation

28 May

VassiliosA guest post from Vassilios Paipais, Lecturer in International Relations at the University of St. Andrews, Scotland. Vassilios holds a PhD in International Relations from the LSE and has published in Review of International Studies, International Politics and Millennium, and held various teaching posts at the LSE, SOAS, UCL and the University of Edinburgh. His work focuses on International Relations theory and international political theology. He is also co-founder of Euro Crisis in the Press and Associate to the LSE IDEAS Southern Europe International Affairs programme. You can read his Euro Crisis posts here, as well as follow him on Twitter.

This post is based on a recently published article in Millennium where he explores the implications of post-foundational political ontology for IR via a reading of Martin Heidegger and Oliver Marchart.


Hans Morgenthau

Post-foundational political thought offers the conceptual tools to theorise the experience of dislocation in politics signified by the difference as such between politics and the political. According to Žižek, the political designates the “moment of openness, of undecidability when the very structuring principle of society, the fundamental form of the social pact is called into question” whereas politics describes the positively determined outcome of that process, a “subsystem of social relations in interaction with other sub-systems”. The difference as such between politics and the political implies that any effort to cancel this gap or gloss it over by using ethical, political, juridical or economic arguments is nothing else but an attempt to hegemonise the social by ideologically displacing politics. The political signifies the moment of grounding/de-grounding of the social that is suppressed or forgotten by the operation of politics but can be reactivated at any time through dislocation and antagonism. Politics is incessantly trying to colonise the political but we are each time painfully reminded that an unbridgeable chasm separates the two. It is exactly the irresolvability of this gap that makes politics the name for a paradoxical enterprise which is both impossible and inevitable – which is why none has ever witnessed ‘pure politics’ either. The political cannot be brought about voluntaristically but, whenever we act, it is as if we always activate it or, better, we are always enacted by it. Both gestures of eliminating the force of the political (post-politics) or of introducing it unmediated into politics (total war, revolutionary terror) end up abolishing the political difference and ultimately result in an ideological displacement of politics.

Against this backdrop, I read the sophisticated realism of Hans Morgenthau as a promising but inconclusive attempt at a post-foundationalism political ontology. In fact, I argue that by equally shunning a facile surrender either to the immanence of power (ultra-politics) or to the technologisation of politics (post-politics), Morgenthau’s theory of the political strove to maintain a reflexive fidelity to the logic of political difference as such. At this point, the question naturally arises: why Morgenthau? Isn’t he the archetypical exponent of a tradition that prioritises a static view of international relations and the adoration of power politics? Well, for those who have been following the recent revisionist literature on classical realism, not really; Morgenthau, in contrast, emerges as an apparent candidate to discuss the crisis of foundationalism in (international) political thought and the paradox of its necessity and impossibility, not least because he is one of those rare thinkers that offers no facile solution to, or redemption from, the existential anxiety caused by the interrogation of ultimate foundations in late modernity.

Such an exercise highlights the strong affinities between Morgenthau and critical historicist currents in social and political theory, but this would come as a surprise only to those who equate Morgenthau’s realism with stasis and conservatism and are ignorant of his debt to the thought of Dilthey, Mannheim and Nietzsche. And yet, why inconclusive? Short answer: because of his failure to be radical enough in his Kantian antinomism or, to put it reversely, in his Nietzschean skepticism. And yet, my intention is not to award or withhold credentials of criticality, nor to indict Morgenthau for failing to live up to standards that he never set for himself. On the contrary, in an authentic act of immanent criticism, one does not seek to oppose the other(s) but, instead, to bring out a certain ‘internal contradiction’ to them, in a sense repeat all that they are saying but for an entirely different reason. The purpose of this critique is not to identify shortcomings in Morgenthau’s arguments but to interrogate the ‘transcendental’ conditions of his discourse: that which is in it more than itself. My thoughts on Morgenthau’s unfinished project then should be seen as a propaedeutic towards an investigation of the conditions and challenges involved in practicing international theory as a constant critique of depoliticisation.

Continue reading

Aircraft Stories: The F-35 Joint Strike Fighter (Part II)

17 May

This is the second part of a single post about the F-35 as actor-network. The first part is here 

wbbuca 013

Strike

This word is meant to convey the F-35’s identity as a proper multirole fighter, a machine rigged to conduct both air superiority and strike missions, the latter defined as tactical attacks on a ground or naval target with a particular focus on “initial blow” or “first day of attack” operations.  All three variants of the F-35 fighter family hold this capability: the conventional A version designed for use by the U.S. Air Force and allied air forces; the Short Take-Off, Vertical Landing (STOVL) or B variant for the U.S. Marines Corps as well as the UK’s Royal Navy, as well as the conventional carrier-based edition for the U.S. Navy, the F-35C.

Airstrike, or strike for short, shapes, and is shaped by, the evolving structure of international politics in important ways.  Pax Americana, defined in terms of successive hegemonic or hierarchical international and regional orders centered on Washington, D.C., can be regarded as an assemblage made possible by the so-called global strike, among other smaller assemblages.  Since the middle years of the twentieth century warplanes have transformed themselves into multirole, fighter-bomber machines capable of ever-greater lethality and survivability.  What makes U.S. strike aircraft especially formidable is the surrounding stuff—assets like ballistic and cruise missiles plus countless “force enablers” such as ground bases, aircraft carrier groups, logistics depots, a large tanker force and aerial refueling know-how, interlinked information and communication systems, the ability to generate and sustain the use-of-airspace deals on relatively short notice and so on.

Continue reading

Aircraft Stories: The F-35 Joint Strike Fighter (Part I)

16 May

bourget

How big is the F-35 Joint Strike Fighter? By one set of measures, it is three times bigger than the Eisenhower Interstate Highway System, ten times bigger than either the Apollo Project or the International Space Station or Hurricane Katrina, or one hundred times bigger than the Panama Canal. These comparisons are only moderately outlandish. US$1.45 trillion is the Pentagon’s own December 2010 estimate of lifetime operating and supporting costs for the 2,443 copies of the F-35 currently on order by the United States government, which we can then compare to the known price tags, in 2007 dollars, of these five projects.[i] Costs—also variously prefaced as procurement, actual, sunk, fly-away, upgrade, true and so on—and their contestations are central to a discourse of accountancy that surrounds all projects that require large-scale mobilization of public power. But enormous as they are, these numbers still cannot capture the size of this particular weapons program. To understand just how big the F-35 is, I wish to suggest in this two-part post, we ought to conceive it as a proper assemblage—a heterogeneous association of human and nonhuman elements that is at once split, processual, emergent, and, most importantly, constitutive of the modern international.

Continue reading

Follow

Get every new post delivered to your Inbox.

Join 3,221 other followers

%d bloggers like this: