Friday, December 20, 2013

Grants and the measure of a scientist's 'worth'



All academic scientists worry about grants. In my weekly lunches with other faculty, that is often one of  the first points they raise as a source of concern. I anxiously await now a score (if any) from my most recent NIH submission. Some colleagues of mine reported either euphoria associated with their recent NSF funding or sadness with recent NSF declines. Part of the worry is about keeping the science going that we want to do. Part is about not laying off good employees. But sadly, a big part is "image" to the university and to peers. Administrations (and often peers) ask frequently about how big particular grants are and then implicitly or explicitly rank the faculty member based on that amount.

We use dollar amounts of grant funding to assess faculty all the time: for hires, yearly performance raises, tenure and promotion, etc. Part of this is reasonable-- grant funding is competitive, so in an ideal world, someone who has good, creative ideas for feasible, high-impact research should acquire grant funding more easily than someone who lacks such ideas. Further, virtually no research is "free"-- one needs salary for themselves and their labs. That research may be subsidized by the university (as part of our "9-month" salary and TA-ships for students), but it's still not free. Hence, we need money to sustain our research.

That said, many readers will agree we've become too grant-obsessed in our assessments at all levels. New faculty members are immediately dubbed "successful" or "hotshots" if they acquire funding early, whereas early publication of high-impact research in a big journal often has a lesser effect. I recall once (I'll be vague for confidentiality) when 2 assistant professors were up for tenure simultaneously. One had multiple papers in the most premier science journal and multiple others elsewhere, and with a consistent publication rate along his years, but had acquired very little funding. The other acquired federal funding early but didn't publish anything until they put out a small number of papers in the year they were up for tenure (and none is journals as prestigious as the first). The faculty tenure vote was more strongly favorable for the second than the first, citing "sustainability" as a concern for the first.

Let me use an analogy. Funding is like gas to make the engine of research run. However, comparing faculty based on grant dollars is like comparing two cars in how far they'll go based on how much gas is in their tank. Many scientists are like Toyota Prius plug-ins (beyond their interest in reducing emissions)-- yes, they need gas, but they can go very, very far with a small amount (~58mpg). Other scientists may be more like an 8-cylinder Chevrolet Camaro (~14mpg), or even a coach bus (~6mpg). There is even empirical evidence that very heavily funded labs, on average, produce less per dollar than mid-sized labs (ref).

Again, research isn't free, and sustainability is a concern, so we should not ignore funding. However, I will argue that IF we are to use grant dollars as part of a measure of evaluation, we should simultaneously consider that investigator's track record of "research efficiency per dollar" (like gas mileage). How many impactful new insights have they published per dollar? Shouldn't we be in awe of those who publish high impact work while using up less taxpayer money (and thus leave more for other research)? Shouldn't we consider research sustainability not just by how much you have, but how well you'll do in the inevitable situation in which you have little research funding? There are multiple ways to publish in PLoS Biology, Science, or Nature--  two are "scale" (you do something that's slightly creative but on a grander scale than anyone else has done before-- clearly expensive) and "innovation" (you come up with a truly creative idea and test it-- perhaps not expensively). It's time we spent more effort giving attention and reward to the latter of those two approaches, especially now when grant dollars are preciously limited.

5 comments:

  1. I really like the 'research efficiency per dollar' approach. In fields that don't involve molecular techniques, the cost of running the lab may be less, but the PI should not be penalized for doing more with less. Also, many young scientists, myself included, are aware of the difficulty in attaining funding. Therefore, we are picking up computational tools and doing more collaborative work so that we don't need as much funding to be productive in those crucial pre-tenure years. I'd like to think that this strategy of investing in increased research efficiency will pay off in the long term, but not if tenure decisions are made on funding and not output. What happened to the mantra 'publish or perish'? Is it now 'prove your worth to a credible funding agency or perish'?

    ReplyDelete
  2. You make some very strong points, and I agree that funding often plays too strong a role in faculty evaluations. If I could play devil’s advocate, however, I would raise two points. First, measuring research impact is notoriously difficult, and I would hesitate to rely on for-profit journals as a critical part of that assessment. It drives the scholarly community towards style over substance. Second, I would argue we need much more time to measure research impact that we usually give ourselves. For instance, in mathematics it is not uncommon to see a “low-impact” theory sit on the shelves for decades before a new technology comes along which using that theory and suddenly increases that impact. I think the hope is that grant review panels conduct nuanced assessments, caring more about scholarship than magazine sales and trying to project the long-term (so far unrealized) impact of a project.

    ReplyDelete
    Replies
    1. Excellent points-- I agree 100%. I wouldn't even consider them devil's advocate per se, as they don't contradict what is said above. Definitely impact is hard and slow to measure (which is, sadly, why many fall back on "metrics" like pub #, grant $, impact factor), and definitely publication in Science/ Nature is not the end or measure in itself, and indeed often reflects flash over substance.

      Delete
  3. My apologies in advance for perhaps being an ignorant outsider* completely putting my foot in my mouth by commenting on a career area unfamiliar to me. But, can I ask whether this blog post ("Grants and the Measure of a Scientist's Worth") and your previous blog post ("Grades: What are they Good For?") were published in close proximity on purpose?

    From an outsider's point of view, there are parallels between the two topics.

    -Related to your comments on using grants as a measure of a researcher's "sustainability": When medical schools (or prospective employers or other graduate schools) look at a student's grades, are they, in some sense, using them as a predictive measure of whether the student will be "sustainable"? (For example, will the student be able to flourish in a demanding, fast-paced medical school or work environment?)

    -For many undergrads, grades are seen as strongly related to their ability to achieve the next stage in their personal and career goals. How much does the "grades" mindset persist as a student progresses up the academic ladder and becomes a researcher in an academic setting? Not to discount academic and economic realities, but is there a sense in which "grades" are directly traded for "grants" in some corner of the academic psyche?

    -Again, not to discount academic and economic realities, but do both grades and grants become, in some sense, a short-term proxy for judging a much more ambiguous, long-term, difficult-to-measure, and worthwhile goal? Perhaps the real goal is something like "intellectual development for life" for an undergrad and "impact of research in posterity" for researchers? How much does this short-term proxy cause more worthwhile long term goals to be pushed into the background or forgotten? How much is this short-term proxy beneficial in providing some tangible motivation to keep putting one foot in front of the other in pursuit of the long term goals?

    -Have you, by any chance, been reading (Stanford mathematics professor) Kenneth Devlin's blog (mooctalk.org) on his experiments with his MOOC on the Coursera platform? Most recently, he has been writing a series on the role of failure in the intellectual development of a student. He is thinking about methods to attempt to break through his MOOC students' instinctive focus on grades and get them to risk failure in order to work towards more important intellectual goals. (What better place to risk failure than a MOOC?) As someone who spent my career in the business (engineering) world, I see the attractiveness and benefit of a concept like "research efficiency per dollar," but it also makes me a bit queasy, thinking back to how similar concepts sometimes work out in practice in the business world. My thoughts are very vague at this point, but I wonder about how the metric of "research efficiency per dollar" (inevitably sometimes poorly applied in the real world) would interact with the concepts of risk-taking, failure, and intellectual achievement.


    *I started reading your blog because I was a Duke student in ancient times, am currently interested observing in the evolution of MOOCs, ran across a link to your blog on the Duke Center for Instructional Technology site, and have found your blog thought-provoking.

    ReplyDelete
    Replies
    1. Great thoughts on all counts-- no parallel between my two posts was intended, but your points are well taken. I especially like your third point about proxies... grades are definitely a proxy for "something" (be it understanding or effort or intellectual ability or some combination of those) and one function of consideration of grant dollars is as a proxy for sustainability or potential impact of the proposed research.

      Delete