Category Archives: Peer Review

Catalogue

I have now written enough posts in this blog that it needs a catalogue. I have organised the posts into 8 themes:-

  1. How to write a Grant Application
  2. Strategy for writing grant applications
  3. Writing Style for Grant Applications
  4. Giving and Receiving Feedback on Grant Applications
  5. Dealing with referees reports and with rejection
  6. Interviews and Talks
  7. Software
  8. Academic Life and Afterlife

How to Write a Grant Application

Strategy for Grant Applications

Writing Style for Grant Applications

Friendly fire: Giving and receiving feedback

Dealing with referees reports and with rejection

Interviews and talks

Software for Writing Grant Applications

Academic life & Afterlife

Share

Ten questions to Get Feedback on a Grant Application

FeedbackThis post was triggered by an  article in Harvard Business Review about how to get feedback on your performance at work. Getting feedback that you can trust is a universal problem and it affects grant-writers more than most. The key to getting good feedback  on a grant application is to think carefully about whom you should ask and how you should ask them.

Whom should you ask for feedback on your grant application?

The best option is to ask the right kind of expert, if you can. Expertise on grants is much more important than expertise in your subject. The ideal person would be someone who has served several years on the grants committee that will consider your proposal. Failing that, you could ask someone who has wide experience of similar grants committees and a good understanding of the scheme for which you are applying.  Unfortunately these people tend to be busy and you may be in a hurry. You can ask Parker Derrington Ltd to read your grant application and give you face-to-face feedback by Skype or Facetime, but you would have to pay. What can you do if you don’t have an expert on call and can’t afford to pay one?

If you can’t get an expert, you should choose an ignoramus, or someone who can act the part as far as your project is concerned. You need someone who does not share your views about what is important in your subject but who is open-minded. This mimics the kind of expertise that a grants committee member will have. The ideal person would be someone at your level, in a different but related field. It can be a good idea to exchange feedback with a colleague in a different subject area who is also preparing a grant application. It is a mistake to use someone who knows as much or more than you do about the area of your project, unless they are sufficiently experienced in the way grants committees work that they will be able to discount their knowledge of your subject.

How should you ask for feedback on your grant application?

If you ask a real expert for feedback, it doesn’t matter how you ask, they will give you useful feedback. Otherwise, you need to design your questions carefully if the answers are to be useful. Here are some suggestions.

  1. Write down the main outcome of the project. Mark where it describes it in the text with a (1)
  2. Mark with (2) in the text the section that gives evidence that the main outcome is important. Can you think of anything else that could be said to support the claim that the outcome is important?
  3. Is the importance of the outcome specific to this funder? Mark any references to the funder’s goals with (3).
  4. How many separate components does the research project have?
  5. Mark with (5-1), (5-2) etc, the sections of text that state the outcome of each component of the research project.
  6. Mark with (6-1), (6-2) etc, each section of text that explains how important the outcome is of a component of the research project.
  7. Mark with (7-1), (7-2) etc, the sections of text that give citations that support the explanations in (6-1), (6-2) and so on.
  8. Mark with (8-1), (8-2) etc, the sections of text that describe the research processes that will produce the outcomes (5-1), (5-2) etc.
  9. For each of these sections, can you mark with (?) any processes that are not clear, and can you suggest any extra processes that might be needed in order to produce the research outcome.
  10. Mark in the text with (1) where it states what will be done to make sure that the overall outcome of the project has maximum value to the funder. Can you think of any way that this could be made more valuable to the funder.

You should ask the reader to spend no more than half an hour answering these questions. If they are unable to answer them in that time then you need to do some serious rewriting. You might also consider reading chapter 11 of The Research Funding Toolkit, which has a comprehensive set of exercises to get feedback at every stage.

Finally, you should know that the correct answer to question 4 should be three.

Share

We thank the referees for their helpful suggestions…….

PeerReviewCartoonIt may not be easy to keep your temper when responding to referees’ comments but you must make it seem like the easiest thing in the world. Whether the comments are on a grant application or a paper, you should compose your response with regard to the effect you want it to have on the reader. If you want your response to influence a grants committee to award your grant or a journal editor to publish your paper, anger is unlikely to help.

There are some differences between responding to grant-application referees and paper referees but most of the following recommendations apply to both.

Assume that the referee is trying to be helpful. You will do a better job if you can make the assumption of reasonableness. If you can convince yourself that the referees are doing their best to help you  and their comments are genuinely trying to help you improve your research project or your paper, it will be easier to follow most of the recommendations below. It doesn’t matter if the assumption is wrong and the referees are actually trying to sink your grant or turn your paper into tedious gibberish. The important point is that you will do a better job of responding to them if you assume that they are trying to help.

Take responsibility for the reader’s failure to understand. Some writers take pride in the knowledge that very few people understand their work. I have seen responses from grant applicants that berated referees for their stupidity. That is a risky attitude to take in any type of writing and is almost certain to lead to the failure of a grant application.

Express gratitude for referees’ suggestions. Derrington’s first law of responding to referees states “The less you feel gratitude, the more fulsomely you should express it”. This is probably more important in the case of a paper than a grant application because in the case of a paper the referee will usually be consulted about whether your response is satisfactory whereas in the case of a grant application your response will be usually assessed by the grants committee.  Of course it may help you to express gratitude if you can explain how the referees insightful suggestions have enabled you to transform the research project or paper.

Make it clear that the responding to the referees’ suggestions has enabled you to improve the research project (or paper). There are two reasons for this.

  • First, you want to give the impression that the final version of your paper (or grant application) is much better than the version that the referee evaluated and deserves a higher score than the referees gave it.
  • Second, you may want to distract the referee (or the grants committee) from the fact that you may not have made any changes at all. In this case, Derrington’s second law of responding to referees applies “The less you change, the more emphatically you state how much you have changed and how much this has transformed the paper.”

Derrington’s second law is more important for papers than for grants, because journal editors often regard referees’ recommendations as binding and authors often find them unacceptable. A grants’ committee is much more likely to recognise when a referee’s recommendation is ridiculous and accept a response that politely declines to implement it.

State clearly what you are responding to and how. This is particularly important when referees’ comments and suggestions are vague or ambiguous. It is often helpful to paraphrase the referees’ suggestions and to state what changes you have made in response to each one. This is a good way of dealing with the case when two or more referees suggest almost exactly the same thing. A list of your paraphrasings of the referees’ suggestions and a short statement about how you have responded to each one can make a very helpful executive summary that will reassure the reader that you have responded satisfactorily.

Some journals require you to do this but few funding agencies do. Even so, you should do it because it is likely to increase your chance of getting funded because it reassures the committee that you have responded satisfactorily to the referees’ criticisms.

Keep your overall response as short and simple as possible. This is more important with grant applications than with papers because the committee works under immense time pressure.

Know when to give up. If the referees’ reports on a grant are uniformly lukewarm it is unlikely to get funded whatever you say. Paradoxically, faint praise is more damning than strong condemnation because condemnation can give the impression that the referee is biased.

Of course, if you are overwhelmed by the desire to let the referees know how stupid their comments are, you probably won’t be able to follow any of the advice I have given so far. In that case I suggest that you wait until you can. Responding to referees is like inviting an  elderly relative to dinner, better not to do it than to do it with bad grace.

Share

The Summary: Your direct line to committee feelgood.

18534682_sNot many people realise that the summary section on your grant application is a direct line to the most influential member of the committee that decides where it sits in the funding priorities. You can make this person feel really good about your application, just as they start to read it. Let me explain how it works.

Grants committees everywhere are overloaded, thinly stretched and work very fast. Committees I have worked on would decide on a £300K grant in less than 10 minutes. Sometimes a lot less. I can remember a panel that decided on 72 applications in 6 hours. Meetings commonly last two days, so committee members will often have more than 100 applications to read for a meeting.

The subject spread of the applications is an even bigger problem than the numbers. Everybody on the committee is an expert on something. But expertise tends to have a very narrow focus, and for any individual member, most of the applications are outside that focus. Committees cope with this by designating a member, sometimes two or three, to become an expert on each grant. It is the job of these designated members to present the grant application to the committee.

The designated member is crucial to the success of your application. They explain to the committee what the grant is about. They say what the applicants propose to do, what are the specific research aims, how the proposed research project will meet those aims, and what will be done with the results. They summarise the referees’ recommendations and they recommend a score. It takes about 5 minutes. This presentation is hugely influential. Most of the committee will go along with the recommendation.

This system, or something like it, is widely used in the UK and overseas. There’s an excellent video illustrating how the US National Institutes of Health does it on youtube.
https://www.youtube.com/watch?v=fBDxI6l4dOA&feature=youtu.be and a blog describing what it feels like for a junior academic to participate in the process here.

The nature of the review process means that the summary of your grant application has a huge influence on the designated member who presents your grant. Let me put you inside the head of a one-time presenter.

Presenting a grant is not easy. It’s always a stretch to get your head around someone else’s research ideas. There is a lot to keep straight in your head. And you are presenting in front of colleagues whose respect is important to you. You don’t want to look as if you are out of your depth in front of them. You feel that you want to do a good job.

Actually, you have to do more than one good job. Usually you have to prepare several presentations and keep them all straight in your head. Then you present them as each grant comes up. I once had to do 12 in one day.

As you pick up each application on which you have to prepare to speak, you have in mind both a nightmare and a dream. You can probably guess what the nightmare is. I want to tell you about the dream, because you have the chance to make it come true.

The dream is that the summary, which is the first thing you read, will start by saying exactly what problem the applicant proposes to solve and how. Then it will say what it is that makes the problem important, what are the specific research aims, how the proposed research project will meet those aims, and what will be done with the results. In short, the designated member’s dream is that the summary would be the ideal set of notes for a talk to explain the grant to the committee.

The dream continues: the introduction to the case for support makes exactly the same statements as the summary. Then the remainder of the case for support convinces the reader that the statements are true with detailed, evidence-based, argument and explanation.

So think about this dream as you write the summary of your next grant proposal. Make the dream come true. It will give your grant a huge advantage in committee.

Share

Review a Research Grant Application in Five Minutes

PeerReviewCartoon

Cartoon by Nick D Kim, http://strange-matter.net

This post tells you how to do a five-minute review of a research grant application. If you are asked to comment on a grant application by a friend or colleague, you should begin with this five minute review. In 95% of cases it will be all you need to do. Except create the feedback sandwich of course.

If you are writing a grant application, before you ask anybody else to read it you should spend five minutes reviewing it yourself. Far too many of the grant applications that I get asked to read take me a lot less than five minutes to review. Then it takes several days to construct a palatable feedback sandwich with the filling  “rewrite this completely”.

I don’t say you can review the detailed content of a grant application in five minutes. That takes longer and I will write about how to do it quickly in a future post. However, five minutes is plenty of time to review the framework of the case for support and check that it is appropriately used.

The framework is important for two reasons. First, if it is good, it tells the reader the essential story of your grant application very quickly. Remember, most readers only want to know the essential story. Second, the framework guides the reader to the detailed content that supports and justifies the essential story, so that the detail can be reviewed effectively and quickly.

‘Summary Sentence’

The most important part of the framework is a summary sentence. This should say what the project will achieve and enough about how it will achieve it to give a bit of distinctiveness and a bit of plausibility. It is essentially the elevator pitch, except that instead of taking 30 seconds to 2 minutes to say it or read it, it should take more like 15 seconds. The summary sentence should be the first sentence of the introduction, so it should take no time to find it, you still have 4 minutes 45 seconds left.

If you read my last post you will recognise that the summary sentence is what I refer to there as the first key sentence. So it will come as no surprise that you should check that the summary sentence is re-used as the first sentence of the part of the case for support that sets out the background to the research project (In line with previous posts  I am going to refer to this as the “Background” although different funders call it different names). Let’s allow 15 seconds to do that. 4 minutes 30 seconds left.

‘Importance Sentence’

Now you should spend 20 seconds checking the second sentence of the introduction. It’s the importance sentence. It should say something about why it is important to achieve whatever the project will achieve. Make a mental note about whether the importance sentence has a practical element. Does it mention a real world problem – childhood cancer, the economy, forgetfulness in old-age, or some such. It’s not essential that there is a practical element to the importance sentence but it has implications for the dissemination sentence, which you will review in a few minutes time.

As soon as you have checked the importance sentence for practical content, jump to the background section and check that the importance sentence is repeated there. This is just a quick glance, so it should only take you 10 seconds. You have 4 minutes left.

‘Aims Sentences’

Go back to the introduction. Immediately after the importance sentence there should be a set of aims sentences, about four sentences setting out the aims of the project. The aims sentences should state things that we need to know, understand, characterise or in some way discover. The aims sentences could be preceded by a linking or framing sentence and they could easily be formatted as bullet points. Checking that the aims sentences are in the right place should take you 10 seconds. Do not look at the content of the aims sentences yet. Wait 30 seconds until we get to the sub-project overview sentences.

Now you should check the next sentence of the introduction. It should be the ‘project overview sentence’.

Project overview Sentence.

The project overview sentence gives a simple one-sentence overview of the project. It might also be structured as a linking sentence to the four or so sub-project overview sentences, which should immediately follow it and which could appear as bullet points and could be expressed as objectives. It should take you 30 seconds to read the project overview sentence and 20 more to check that it appears again to introduce the part of the case for support that gives the detailed description of the research methods and of the project.

As Jacqueline Aldridge points out in the Research Funding Toolkit blog,  the overview sentence often gets buried at the end of the section that it should introduce.

Next you can spend 10 seconds checking that the sub-project overview sentences follow the project overview sentence. Now you are ready to check their content against the content of the corresponding aims sentences.

Sub-project overview sentences; content of aims sentences

Each aims sentence should link logically to the corresponding sub-project overview sentence in the following way. The aims sentence should say “We need to know X”. The sub-project overview sentence should give a very brief description of the research in the sub-project and should end with a clause that says “this will tell us X“.

The aims sentences should be repeated in the background. Each of them should introduce a discussion of the evidence that supports the assertion that this is an important aim.

The sub-project overview sentences should be repeated in the part of the case for support that describes the project in detail. Each of them should introduce the detailed description of the corresponding sub-project.

This checking and matching will probably take nearly a minute to do for the first aim/sub-project overview sentence pair but the others should be much quicker. It’s just a matching exercise after all. Lets assume it takes 2 minutes. You have a minute left, and only the dissemination sentence to check.

The dissemination sentence

The last sentence of the introduction should say what will be done with the results. It is the dissemination sentence. You need to check that it is repeated in the description of the project as the introduction to a section on dissemination. You also need to check that, if the importance sentence has a practical component then the dissemination must have a corresponding practical element. For example, if the project is going to discover a cure for a disease, the dissemination needs to promote the use of the cure in some way. As a general rule there must be a plan to disseminate the results in a way that makes it likely that the claimed practical benefit will be realised.

What to do with the outcome of the review

Only about 5% of grant applications will pass this five minute review. If you are reviewing a grant application for someone else and it passes, it should be pretty easy to review the detailed content. I’ll give some helpful pointers in a future post. It goes without saying that if the grant you are reviewing is yours, it is safe to give it to someone to review.

In the more likely case that the grant application fails the review, then it needs rewriting. If it’s yours, that’s pretty simple. Read my post from last week. Write the 12 key sentences and use them to organise the text you have already written. If you can’t write the 12 key sentences then you shouldn’t be writing a research grant.

If the grant application is not yours, you need to write a feedback sandwich. Basically the grant needs to be rewritten before it’s worth looking at the detailed content. I’ll say something about how to do this in my next post but whatever you do don’t use the phrase “completely rewrite”.

 

Share