'Tis the season. No, not that one- like every year at this time (and two other times each year- once in May and once in September), many of us get a stack of grant proposals to review. I just got my stack yesterday and haven't glanced at them yet. For those of you outside the science research community, researchers spend months preparing these proposals (12-15 pages in length, plus several more pages of budget, resume, and other nuances). They agonize over minutia like whether to cite X or Y paper, whether to talk about contingency plans in detail or to present an air of confidence in the approach, whether to present the fine experimental details or save space for more justification on the value of the study to the field, etc.
After these months of work, three or more other anonymous scientists ("reviewers"/ "panelists") read over the proposals- rarely spending more than a few hours on any one in particular (and frighteningly sometimes less). They write a critique- strengths and weaknesses of the work (kudos to NIH for forcing reviewers to actually mention the strengths). The work is discussed in a meeting with ~20 other scientists who have not read the proposal. And finally, it's given a score. Then, in large part using that score, the funding agency decides whether to fund the research or not. Odds of funding are around 10% (don't nitpick- this is close enough).
Most of the ones I read are from people like me- university professors. So, my question is this- is this really a good use of our time? And if not, is there a viable alternative, or is this "the best of all evils"? Let's think of how many person-hours are used in proposal preparation and proposal review. Some of it is definitely worthwhile- forcing the scientist to carefully plan his/ her research and think of what would be the highest impact rather than status quo. The presentations often help guide the subsequent publications in thoughtful ways. Similarly, reviewers benefit by getting a broader view of the field and what kinds of research people are doing.
But there are downsides, even beyond the time commitments of all the parties involved. We all know a few outstanding scientists who seem incapable of writing decent proposals for many different reasons. We also all know that there are some who are great "salespeople" but may be lacking on scientific follow-through or rigor. The present system strongly favors the latter over the former. There's also a general "risk aversion" that comes across in peer review, despite extensive efforts by funding agencies to prevent this. But, if only ~10% of proposed research will be funded (for reference, in a two-day grant review meeting, this may be ~6 proposals), reviewers sometimes have a hard time suggesting that something that "would be very cool but might not work" get funded over something that "would be kind-of cool but will for-sure work." Peer review is also far from perfect- many good proposals get sunk because a reviewer misunderstood something or had a different perspective on what is useful than the proposer. Finally, and frighteningly, university promotion and tenure decisions often seriously consider funding success- seeing it as a quick surrogate to how the community views one's research.
Scientific publishing has gone through major revolutions in the past decade, first with a push for "open-access", and more recently a decision that the merits of science should be sometimes judged historically (the essence of PLoS One): all work that is scientifically valid should be published and history will decide what was most useful. Obviously, public funds (remember- the public funds all this with taxes!) are more precious than whether something is published or not, so a direct analogy is not possible. But is there another way? Or is this really the best of all evils?
