Tag Archives: Referees

What’s the Point?

I want to explain why I think it’s better to produce a well-written grant application than a poorly written one. Obviously, given the nature of my business, I have to make this case, but it is not as simple as you might think, not least because most successful grant applications are very poorly written. In fact, if you think carefully about the quality of grant applications, it becomes clear that, in this particular domain, quality is completely subjective. So I will start by saying what I think makes a good grant application.

The quality of a grant application is not the same as the quality of the research project it describes. A grant application is essentially a marketing document for a research project and you can have a first-rate application that markets a tenth-rate project. And vice versa. Indeed  poorly-written grant applications are very often successful precisely because grants committees are trying to judge the quality of the project, not the quality of the application. Judging the quality of a project can be very difficult if the application is poorly written. So what makes a good grant application?

The essence of a good grant application is that it makes it easy to judge the project. The application contains all the detail that an expert will look for. The detail should be set out so that it can be read at very high speed and understood by a non-expert. As a rule of thumb, it should take less than two minutes to understand the main points of what you will do and why it is worth doing.

Those main points should be expressed and justified in such a way that a non-expert ‘gets’ what you are going to do and why. An expert should also be able to drill down and find the detail that they need in order to judge whether your project is likely to succeed and achieve those main points. I have already explained how the ‘key sentence’ structure enables a grant application to fulfil these requirements.

Despite the fact that most successful grant applications are poorly written, there are three reasons that it is worth taking trouble to produce a well-written grant application:-

  1. If your project is good, a well-written application will improve your chance of success.
  2. If your project is bad, a well-written application will help you to see that it needs to be improved.
  3. A well-written application can be easier and quicker to write than a badly-written application.

I’ll deal with the first two reasons in this post and I will leave the third for the future.

Well written applications are more likely to be successful.

Well-written applications generate an enthusiasm among committee members that makes them give higher scores. For reasons I’ll explain in a future post, the person leading the discussion is likely to recommend a relatively conservative score, no matter how much they like the application. But if the committee are enthusiastic, they are quite likely to argue that the recommendation should be raised, and to exceed the recommendation when they score.

Poorly written applications can also get high scores, particularly if the referees have given very strong recommendations, but when committee members don’t understand an application they will not argue for a higher score and they may even score slightly below the recommendation. The consequence is that the scores of poorly written applications tend to drift downwards. The effect is small, but if the score is close to the borderline, which is likely to be the case, given the tendency for conservative recommendations, a tiny drift can make the difference between success and failure.

A well written application helps you see that you need to improve your project.

A well-written application explains your project very clearly at two levels.

  • First it explains what makes the project important to the funder.
  • Then it explains what the project consists of, and why each part of the project is important.

If your project needs to be improved, you are likely to find one or both of these explanations unconvincing as you write them. If you do find yourself writing arguments that you find unconvincing, then you need to reexamine your project and work out how to make it more convincing. If your application does not convince you, it is unlikely to convince a committee.

 

Share

We thank the referees for their helpful suggestions…….

PeerReviewCartoonIt may not be easy to keep your temper when responding to referees’ comments but you must make it seem like the easiest thing in the world. Whether the comments are on a grant application or a paper, you should compose your response with regard to the effect you want it to have on the reader. If you want your response to influence a grants committee to award your grant or a journal editor to publish your paper, anger is unlikely to help.

There are some differences between responding to grant-application referees and paper referees but most of the following recommendations apply to both.

Assume that the referee is trying to be helpful. You will do a better job if you can make the assumption of reasonableness. If you can convince yourself that the referees are doing their best to help you  and their comments are genuinely trying to help you improve your research project or your paper, it will be easier to follow most of the recommendations below. It doesn’t matter if the assumption is wrong and the referees are actually trying to sink your grant or turn your paper into tedious gibberish. The important point is that you will do a better job of responding to them if you assume that they are trying to help.

Take responsibility for the reader’s failure to understand. Some writers take pride in the knowledge that very few people understand their work. I have seen responses from grant applicants that berated referees for their stupidity. That is a risky attitude to take in any type of writing and is almost certain to lead to the failure of a grant application.

Express gratitude for referees’ suggestions. Derrington’s first law of responding to referees states “The less you feel gratitude, the more fulsomely you should express it”. This is probably more important in the case of a paper than a grant application because in the case of a paper the referee will usually be consulted about whether your response is satisfactory whereas in the case of a grant application your response will be usually assessed by the grants committee.  Of course it may help you to express gratitude if you can explain how the referees insightful suggestions have enabled you to transform the research project or paper.

Make it clear that the responding to the referees’ suggestions has enabled you to improve the research project (or paper). There are two reasons for this.

  • First, you want to give the impression that the final version of your paper (or grant application) is much better than the version that the referee evaluated and deserves a higher score than the referees gave it.
  • Second, you may want to distract the referee (or the grants committee) from the fact that you may not have made any changes at all. In this case, Derrington’s second law of responding to referees applies “The less you change, the more emphatically you state how much you have changed and how much this has transformed the paper.”

Derrington’s second law is more important for papers than for grants, because journal editors often regard referees’ recommendations as binding and authors often find them unacceptable. A grants’ committee is much more likely to recognise when a referee’s recommendation is ridiculous and accept a response that politely declines to implement it.

State clearly what you are responding to and how. This is particularly important when referees’ comments and suggestions are vague or ambiguous. It is often helpful to paraphrase the referees’ suggestions and to state what changes you have made in response to each one. This is a good way of dealing with the case when two or more referees suggest almost exactly the same thing. A list of your paraphrasings of the referees’ suggestions and a short statement about how you have responded to each one can make a very helpful executive summary that will reassure the reader that you have responded satisfactorily.

Some journals require you to do this but few funding agencies do. Even so, you should do it because it is likely to increase your chance of getting funded because it reassures the committee that you have responded satisfactorily to the referees’ criticisms.

Keep your overall response as short and simple as possible. This is more important with grant applications than with papers because the committee works under immense time pressure.

Know when to give up. If the referees’ reports on a grant are uniformly lukewarm it is unlikely to get funded whatever you say. Paradoxically, faint praise is more damning than strong condemnation because condemnation can give the impression that the referee is biased.

Of course, if you are overwhelmed by the desire to let the referees know how stupid their comments are, you probably won’t be able to follow any of the advice I have given so far. In that case I suggest that you wait until you can. Responding to referees is like inviting an  elderly relative to dinner, better not to do it than to do it with bad grace.

Share

Committees and Referees

Committees Like a Simple Story: thanks to Science and Ink http://www.lab-initio.com

Committees prefer a clear story: http://labinitio.com

The Journal Nature reported yesterday that scientists have complained that there is a mismatch between expert referees’ evaluation scores of research grant applications and funding decisions.  Different interviewees claimed that this mismatch either does or does not indicate either a flaw in the system or mistakes by referees or by committees.

There may be flaws in the system and mistakes probably happen, but there is a more obvious reason that referees’ scores should not be expected to predict funding decisions. Referees and committees do different jobs which impose different constraints on the way a grant application is written. Very few proposals are written in a way that satisfies both sets of constraints, and so, for the majority of proposals, there is no reason to expect a close match between the referees’ score and the committee’s score. Before I explain the constraints and how to meet them, I’ll clarify the story and explain its relevance.

The story was based on 302 grant applications to the Medical Research Council. It states that some applications that received high scores from the referees were triaged (rejected without being discussed by the committee). Of the proposals discussed by the committee, the story states that the group of applications that were successful and the group that were rejected had ‘a nearly identical spread of scores’. It’s worth noting that the story focuses on a statistic of the scores that is not particularly informative (spread) and does not mention any other statistics. It is relatively common that the referees include both friends and foes of the applicant, which can cause the spread of scores on a single application to range from the lowest to the highest possible. Consequently, nothing that the story says about this rather small data sample indicates a general failure of referees’ scores to predict committee decisions.

However, whatever we may try to conclude from this small dataset, most funding agencies (the EPSRC is an exception in the UK) ask referees and committees to do very different jobs. These jobs depend on different aspects of the way the proposal is written.

Referees work alone and each one works on a single application. The referees are expected to be experts in the same research field as the applicant and their evaluations are typically seen as coming from within that field. Their main task is to test the detailed rationale of the proposal. Are the research questions important? Is the research approach feasible? Does the research team have the ability to carry out the project? What is their standing in the field? Has their previous work made a significant contribution? Is the approach novel? Are other teams likely to get the answers first?

To get a good score from a referee, a grant application must contain relevant detail. Evidence that the questions are important and relevant must be cited. Any evidence to the contrary must be dealt with effectively. The research approach must be described clearly and in sufficient detail to convince a knowledgeable sceptic that the project is feasible, can be carried out with the resources requested, and will lead to the promised outcomes.

The committee works as a collective and takes a view across all the applications before them, typically about 100 for a single meeting. They must also take a view across all the different research fields that the committee supports. They need it to be clear that the project will deliver an  outcome that will have importance beyond the immediate research area and that the applicant has identified an approach that is likely to be productive and that the research team has the skills to deliver it. They need to know what the research aims are and how they relate to the overall outcome. They need it to be clear that the research objectives will satisfy the research aims and deliver the overall outcome and that the results will have appropriate impact

More importantly, the committee also works very quickly. They have to reach an agreed view about grant applications which most or all of them may not understand completely. For the committee to score an application highly they need it to be possible to understand it on the basis of a single hasty reading – or even a quick skim. The case for support should have a very clear structure that states clearly what will be the research outcome, why it is important, what are the research aims, what are the objectives and what will be done with the results.

Of course the best possible case for support has a clear structure that enables the committee to understand it and appreciate its strong points. It also uses the structure to guide the referees to the relevant detail. In my experience cases of support written in this way get very high scores both from referees and from committees. Unfortunately they are very rare but this blog explains in several different ways how to write them.

 

Share