Why Metrics Cannot Measure Research Quality: A Response to the HEFCE Consultation

16 Jun

Pacioli Euclid Measurement

Update 24th June: 7,500+ views, 100s of shares, 200+ signatories! And a new post with some responses to further issues raised.

The Higher Education Funding Council for England are reviewing the idea of using metrics (or citation counts) in research assessment. We think using metrics to measure research quality is a terrible idea, and we’ll be sending the response to them below explaining why. The deadline for receiving responses is 12pm on Monday 30th June (to metrics@hefce.ac.uk). If you want to add an endorsement to this paper to be added to what we send to HEFCE, please write your name, role and institutional affiliation below in the comments, or email either ms140[at]soas.ac.uk or p.c.kirby[at]sussex.ac.uk before Saturday 28th June. If you want to write your own response, please feel free to borrow as you like from the ideas below, or append the PDF version of our paper available here.


Response to the Independent Review of the Role of Metrics in Research Assessment
June 2014

Authored by:
Dr Meera Sabaratnam, Lecturer in International Relations, SOAS, University of London
Dr Paul Kirby, Lecturer in International Security, University of Sussex

Summary

Whilst metrics may capture some partial dimensions of research ‘impact’, they cannot be used as any kind of proxy for measuring research ‘quality’. Not only is there no logical connection between citation counts and the quality of academic research, but the adoption of such a system could systematically discriminate against less established scholars and against work by women and ethnic minorities. Moreover, as we know, citation counts are highly vulnerable to gaming and manipulation. The overall effects of using citations as a substantive proxy for either ‘impact’ or ‘quality’ could be extremely deleterious to the standing and quality of UK academic research as a whole.

Why metrics? Why now? Continue reading

The Onuf Principles

30 May

Nicholas Onuf recently gave an interview over at e-IR. Several people of our acquaintance shared the tail-end of that, in which he is prompted to dispense career advice. Although opinions are indeed like assholes, these are good enough to elevate far above the gutter.


Original image by Stéphanie Saramago

Original image by Stéphanie Saramago

1. Preparing at length for classes does not make you a better teacher. Insofar as it dampens spontaneity, students will think you are boring; this will undercut the self-confidence you thought your lengthy preparations had purchased for you. And, of course, it steals valuable time from your scholarship.

2. Writing is a craft; writing well takes most of us a great deal of work. The usual practice is to think of a problem or issue, formulate a project, do ‘research,’ and then write it up. Bad idea. Keep writing at every stage, even if, in the end, you throw out most of what you have written. Writing makes the problem clearer, points up what more you need to do in the way of research, and, most of all, keeps your writing skills well-honed.

3. Don’t send sloppy, badly crafted papers out for review. As a frequent referee, I see them all too often. Many referees will punish you, not always consciously, for doing so, even if they think you are on to something. Once you think you have a well-crafted piece of work, do send it out, because most referees and editors take their duties seriously and will give you valuable feedback.

4. Be cautious about taking on collaborative projects. We all know that scholarship is a lonely occupation. Collaboration reduces the loneliness quotient and can result in better work than any of the collaborators could have produced on their own. It can also result in a piece of work that no one is entirely happy with. Sometimes collaboration causes damaging tension and bad feelings because of temperamental differences, greater or lesser commitment to the project, and perceived inequities in the distribution of work. All that said, collaborating with my brother on two book projects was hugely rewarding. That it might have been hugely risky never occurred to us.

5. Be even more cautious in participating in symposium projects. Their thematic foci may not match your interests very well; they tend to be superficially refereed and thus are not taken seriously; they also tend to disappear quickly from view. There are exceptions—symposia that mark major developments in the field—but you’ll have a pretty good idea if a particular symposium project has that potential. As a senior scholar, I contribute to symposia because it is fun to do projects with friends and I can afford the luxury. Most of all, avoid editing symposium volumes. This involves collaboration under the most difficult conditions. It is extraordinarily time consuming. Wrangling recalcitrant contributors is too often a thankless and disheartening responsibility.

6. Do not take on too many projects at one time. You will spread yourself too thin, miss deadlines, and make it all the more likely that you will succumb to the 90% rule—you run out of steam when any given project is 90% done and only needs some fine-tuning to be sent off. You will end up with a drawer full of nearly done projects that you have progressively lost interest in and will therefore never finish.

7. Dissertations are apprentice projects, immediately recognisable as such. Turning a dissertation into a book is probably the smart thing to do, but it will often take longer than writing the dissertation did. For most of us, it takes five years to write a good book; World of Our Making took me ten years. Whether you have that much time, institutionally speaking, is another matter.

8. Read every day. When I get up in the morning (early) and get my coffee, I read for 45 minutes. In my case, it has always been something that I do not have to read for whatever I am doing at the time. While this has broadened me immeasurably, for many scholars, a fixed time for reading is an opportunity — perhaps the only opportunity — to keep up on the literature in the field.

9. Whether to jump on a trend in the field’s scholarship, try anticipating a trend, come late to a trend but treat it critically, jump around from thing to thing, or plug away at something few others seem to be interested in is a tricky question, having much to do with temperament. It requires you to ask yourself how ambitious you are, how much you need validation from others, how long you can stayed focused on one thing, et cetera.

10. On the assumption that you are smarter than most people (or you would not be a scholar), seek out people whom you know to be smarter than you in various obvious ways. On the one hand, the more of these people you know, the less intimidating you will find them, and the more you will learn from them. On the other hand, knowing really smart people will remind you of your own limitations and help you be less arrogant. Arrogance is, of course, a constant hazard in our line of work.

Bad Infinity?: Hans J. Morgenthau’s Double Critique of Depoliticisation

28 May

VassiliosA guest post from Vassilios Paipais, Lecturer in International Relations at the University of St. Andrews, Scotland. Vassilios holds a PhD in International Relations from the LSE and has published in Review of International Studies, International Politics and Millennium, and held various teaching posts at the LSE, SOAS, UCL and the University of Edinburgh. His work focuses on International Relations theory and international political theology. He is also co-founder of Euro Crisis in the Press and Associate to the LSE IDEAS Southern Europe International Affairs programme. You can read his Euro Crisis posts here, as well as follow him on Twitter.

This post is based on a recently published article in Millennium where he explores the implications of post-foundational political ontology for IR via a reading of Martin Heidegger and Oliver Marchart.


Hans Morgenthau

Post-foundational political thought offers the conceptual tools to theorise the experience of dislocation in politics signified by the difference as such between politics and the political. According to Žižek, the political designates the “moment of openness, of undecidability when the very structuring principle of society, the fundamental form of the social pact is called into question” whereas politics describes the positively determined outcome of that process, a “subsystem of social relations in interaction with other sub-systems”. The difference as such between politics and the political implies that any effort to cancel this gap or gloss it over by using ethical, political, juridical or economic arguments is nothing else but an attempt to hegemonise the social by ideologically displacing politics. The political signifies the moment of grounding/de-grounding of the social that is suppressed or forgotten by the operation of politics but can be reactivated at any time through dislocation and antagonism. Politics is incessantly trying to colonise the political but we are each time painfully reminded that an unbridgeable chasm separates the two. It is exactly the irresolvability of this gap that makes politics the name for a paradoxical enterprise which is both impossible and inevitable – which is why none has ever witnessed ‘pure politics’ either. The political cannot be brought about voluntaristically but, whenever we act, it is as if we always activate it or, better, we are always enacted by it. Both gestures of eliminating the force of the political (post-politics) or of introducing it unmediated into politics (total war, revolutionary terror) end up abolishing the political difference and ultimately result in an ideological displacement of politics.

Against this backdrop, I read the sophisticated realism of Hans Morgenthau as a promising but inconclusive attempt at a post-foundationalism political ontology. In fact, I argue that by equally shunning a facile surrender either to the immanence of power (ultra-politics) or to the technologisation of politics (post-politics), Morgenthau’s theory of the political strove to maintain a reflexive fidelity to the logic of political difference as such. At this point, the question naturally arises: why Morgenthau? Isn’t he the archetypical exponent of a tradition that prioritises a static view of international relations and the adoration of power politics? Well, for those who have been following the recent revisionist literature on classical realism, not really; Morgenthau, in contrast, emerges as an apparent candidate to discuss the crisis of foundationalism in (international) political thought and the paradox of its necessity and impossibility, not least because he is one of those rare thinkers that offers no facile solution to, or redemption from, the existential anxiety caused by the interrogation of ultimate foundations in late modernity.

Such an exercise highlights the strong affinities between Morgenthau and critical historicist currents in social and political theory, but this would come as a surprise only to those who equate Morgenthau’s realism with stasis and conservatism and are ignorant of his debt to the thought of Dilthey, Mannheim and Nietzsche. And yet, why inconclusive? Short answer: because of his failure to be radical enough in his Kantian antinomism or, to put it reversely, in his Nietzschean skepticism. And yet, my intention is not to award or withhold credentials of criticality, nor to indict Morgenthau for failing to live up to standards that he never set for himself. On the contrary, in an authentic act of immanent criticism, one does not seek to oppose the other(s) but, instead, to bring out a certain ‘internal contradiction’ to them, in a sense repeat all that they are saying but for an entirely different reason. The purpose of this critique is not to identify shortcomings in Morgenthau’s arguments but to interrogate the ‘transcendental’ conditions of his discourse: that which is in it more than itself. My thoughts on Morgenthau’s unfinished project then should be seen as a propaedeutic towards an investigation of the conditions and challenges involved in practicing international theory as a constant critique of depoliticisation.

Continue reading

Aircraft Stories: The F-35 Joint Strike Fighter (Part II)

17 May

This is the second part of a single post about the F-35 as actor-network. The first part is here 

wbbuca 013

Strike

This word is meant to convey the F-35’s identity as a proper multirole fighter, a machine rigged to conduct both air superiority and strike missions, the latter defined as tactical attacks on a ground or naval target with a particular focus on “initial blow” or “first day of attack” operations.  All three variants of the F-35 fighter family hold this capability: the conventional A version designed for use by the U.S. Air Force and allied air forces; the Short Take-Off, Vertical Landing (STOVL) or B variant for the U.S. Marines Corps as well as the UK’s Royal Navy, as well as the conventional carrier-based edition for the U.S. Navy, the F-35C.

Airstrike, or strike for short, shapes, and is shaped by, the evolving structure of international politics in important ways.  Pax Americana, defined in terms of successive hegemonic or hierarchical international and regional orders centered on Washington, D.C., can be regarded as an assemblage made possible by the so-called global strike, among other smaller assemblages.  Since the middle years of the twentieth century warplanes have transformed themselves into multirole, fighter-bomber machines capable of ever-greater lethality and survivability.  What makes U.S. strike aircraft especially formidable is the surrounding stuff—assets like ballistic and cruise missiles plus countless “force enablers” such as ground bases, aircraft carrier groups, logistics depots, a large tanker force and aerial refueling know-how, interlinked information and communication systems, the ability to generate and sustain the use-of-airspace deals on relatively short notice and so on.

Continue reading

Aircraft Stories: The F-35 Joint Strike Fighter (Part I)

16 May

bourget

How big is the F-35 Joint Strike Fighter? By one set of measures, it is three times bigger than the Eisenhower Interstate Highway System, ten times bigger than either the Apollo Project or the International Space Station or Hurricane Katrina, or one hundred times bigger than the Panama Canal. These comparisons are only moderately outlandish. US$1.45 trillion is the Pentagon’s own December 2010 estimate of lifetime operating and supporting costs for the 2,443 copies of the F-35 currently on order by the United States government, which we can then compare to the known price tags, in 2007 dollars, of these five projects.[i] Costs—also variously prefaced as procurement, actual, sunk, fly-away, upgrade, true and so on—and their contestations are central to a discourse of accountancy that surrounds all projects that require large-scale mobilization of public power. But enormous as they are, these numbers still cannot capture the size of this particular weapons program. To understand just how big the F-35 is, I wish to suggest in this two-part post, we ought to conceive it as a proper assemblage—a heterogeneous association of human and nonhuman elements that is at once split, processual, emergent, and, most importantly, constitutive of the modern international.

Continue reading

Open Access, Institutionalised?: Or, Another Reason Why International Relations Is Failing As An Intellectual Project

28 Apr

Soc Sci Tweet

The American Sociological Association (ASA) has announced that it will launch an (as-yet unnamed) open access general sociology journal as soon as possible (this year, maybe next). Its proposed features are a mix of traditional and new: there will be start-up cash and a stipend for Editors, peer-review is to be on the standard, appropriately ‘prestigious’ model (but expedited and light on style corrections), a traditional publisher (SAGE) is involved, authors will retain copyright, there will be no hard copies and therefore no limit on how much can be published in any given time period, all articles (accepted or not) will be subject to a $25 processing fee, and a variable tariff of Article Processing Charges (APCs) will be implemented, from free for scholars from “non-competitive” countries to $100-150 for students and $700 for non-members (for the first 12 months, APCs can also be waived, no questions asked).

The editors at Sociological Science (one of whom we interviewed last month) have noticed that this borrows heavily from their own initiative. Sniping aside, this is surely all to the good. An indication that major academic institutions are, at last, taking open access seriously. Not quite overhauling their systems, but adopting publishing platforms considerably more reasonable than the $3,000 APCs and business-as-usual structure previously threatened. This is an important point, since it supports the claim of some OA advocates that APCs may be financially better for the academy than historical subscription rates (I leave exacting comparisons of costs and the burden of double-dipping during any transition to one side). The problem has always been that the prestige economy (and therefore the social reproduction of universities) is not venue-blind. Low cost APCs in marginal journals are therefore of little help for those still seeking the (shrinking) securities of a formal academic post. But when the reputational power of learned societies is applied, it becomes much easier to envision a world of reputable (and hopefully high quality) open access journals charging APCs at a lower net cost than we currently pay through library subscription models.

The ASA is a powerhouse in these terms, and enjoys more market influence than the International Studies Association (boasting 13,000 members to our 7,000). It is all but inevitable that the mainstreaming of open access in this way will put the squeeze on the smaller open access journals, very many of which are labours of love, and some of which seem to actively treasure their reputation as insurgents or irrelevancies. If we want more material (and particularly the kind of material that carries value in an academic market) to be open access, imitation is the right kind of problem to have. Cultural Anthropology is another example of that shift (we got the gossip from them too last year), funding an open access conversion through the largest section of the 12,000 member American Anthropological Association.

Journal Profits

Profitability data from Harvie et al., 2012.

And yet this scenario is once again an embarrassing one for International Relations, which otherwise likes to imagine itself the most engaged and relevant of disciplines (state power! trade rounds! war and peace!). Continue reading

Decolonising the Anglosphere

23 Apr

Alex at Red Fort

A guest post from Alexander Davis, who is a PhD candidate in international relations at the University of Adelaide, Australia. Alex holds a research MA on India’s international colonial history from the University of Tasmania and his PhD dissertation is a postcolonial examination of India’s relationship with the Anglosphere, supervised by Priya Chacko, Kaniskha Jayasuriya and Carol Johnson. This post provides an outline of his critique of the ‘Anglosphere’, which follows from previous discussion on The Disorder of Things by our own Srdjan Vucetic, and acts as an introduction to the themes of his forthcoming dissertation. Alex can also be found over at Twitter.[1]


Why India? Why the Anglosphere?

The term ‘Anglosphere’ refers to a distinctly murky combination of states, peoples or cultures with an implication of both cultural superiority and closer international relationships on the basis of a shared identity. Even dictionary definitions of the term illustrate the difficult. For some, Anglosphere is based simply on the English language, for others it includes the ‘cultural values’ associated with the political development of Great Britain. Srdjan’s recent book on the Anglosphere shows excellently how the idea of the Anglosphere is rooted in its colonial history and is an expression of Anglo-western superiority. Because of my previous research and teaching interest in India’s colonial history, the idea of the ‘Anglosphere’ struck me as an assertion of cultural superiority and dominance, suspiciously similar to colonial justifications for imperial rule. Once we realize this, India, just as it was central to the British empire, becomes central to understanding contemporary discourse on ‘Anglosphere’. The first question I asked, sensibly enough I thought, was ‘is India in the Anglopshere?’ I have since realized deep inadequacies of this question, which in turn has led me towards to believe in need for a decolonisation of the Anglosphere subject.

800px-Anglospeak_svg

Thinking the Anglosphere through India

In order to understand contemporary Anglosphere discourse, and the position in which India fits within the concept, we first need to understand its historical context. An early form of contemporary Anglosphere debates on India can be in England on the future of the British empire at the turn of the 20th century which turned into a discussion on the concept of ‘Greater Britain’. This idea was to be federation between Britain and her colonies might look like. Some thinkers at this time saw India as central to the empire, and therefore central to any ‘Greater Britain’. Sir Charles Dilke originally began to use ‘Greater Britain’ as shorthand for the British empire as a whole, but later argued it should only be the ‘English-speaking, white-inhabited, and self-governed lands’. Others, such as historian John Seeley, took up the idea, initially including India on the inside as a territory of the Crown. However, later, in the same book, he argued Greater-Britain needed to be racially homogenous, declaring India to be ‘…all past and, I may almost say, has no future’.

Winston Churchill’s work on the ‘English-Speaking Peoples’, itself an echo of Alfred Taylor’s ‘English Speaking Races’, emphasizes the supposed superiority and unity of these peoples. Conservative historian Andrew Roberts has recently taken it upon himself to follow up on Churchill’s work. Roberts cuts down the magnitude of his chosen topic by ignoring English-speakers outside the geographical centers of the Anglosphere. His approach to India and colonial history is revealed by his depiction, and ultimate defense, of General Dyer’s massacre at Amritsar. This is the worst example of ‘imperial’ history I can think of. Roberts goes so far as to defend Dyer from the propaganda of the nasty Indian nationalists. Even the British government no longer defends this event, though on a recent trip to India, David Cameron declined to apologize for it. Roberts defends the massacre even though many of Dyer’s victims were English-speaking. Leaving the victims out of the ‘English-Speaking peoples’ is a final act of humiliation and dehumanization.

Just as the debaters over ‘Greater Britain’ were unsure of where India might fit with the concept, contemporary Anglospherists are unsure of what to do with India. Continue reading

Follow

Get every new post delivered to your Inbox.

Join 3,606 other followers

%d bloggers like this: