We thank the referees for their helpful suggestions…….

PeerReviewCartoonIt may not be easy to keep your temper when responding to referees’ comments but you must make it seem like the easiest thing in the world. Whether the comments are on a grant application or a paper, you should compose your response with regard to the effect you want it to have on the reader. If you want your response to influence a grants committee to award your grant or a journal editor to publish your paper, anger is unlikely to help.

There are some differences between responding to grant-application referees and paper referees but most of the following recommendations apply to both.

Assume that the referee is trying to be helpful. You will do a better job if you can make the assumption of reasonableness. If you can convince yourself that the referees are doing their best to help you  and their comments are genuinely trying to help you improve your research project or your paper, it will be easier to follow most of the recommendations below. It doesn’t matter if the assumption is wrong and the referees are actually trying to sink your grant or turn your paper into tedious gibberish. The important point is that you will do a better job of responding to them if you assume that they are trying to help.

Take responsibility for the reader’s failure to understand. Some writers take pride in the knowledge that very few people understand their work. I have seen responses from grant applicants that berated referees for their stupidity. That is a risky attitude to take in any type of writing and is almost certain to lead to the failure of a grant application.

Express gratitude for referees’ suggestions. Derrington’s first law of responding to referees states “The less you feel gratitude, the more fulsomely you should express it”. This is probably more important in the case of a paper than a grant application because in the case of a paper the referee will usually be consulted about whether your response is satisfactory whereas in the case of a grant application your response will be usually assessed by the grants committee.  Of course it may help you to express gratitude if you can explain how the referees insightful suggestions have enabled you to transform the research project or paper.

Make it clear that the responding to the referees’ suggestions has enabled you to improve the research project (or paper). There are two reasons for this.

  • First, you want to give the impression that the final version of your paper (or grant application) is much better than the version that the referee evaluated and deserves a higher score than the referees gave it.
  • Second, you may want to distract the referee (or the grants committee) from the fact that you may not have made any changes at all. In this case, Derrington’s second law of responding to referees applies “The less you change, the more emphatically you state how much you have changed and how much this has transformed the paper.”

Derrington’s second law is more important for papers than for grants, because journal editors often regard referees’ recommendations as binding and authors often find them unacceptable. A grants’ committee is much more likely to recognise when a referee’s recommendation is ridiculous and accept a response that politely declines to implement it.

State clearly what you are responding to and how. This is particularly important when referees’ comments and suggestions are vague or ambiguous. It is often helpful to paraphrase the referees’ suggestions and to state what changes you have made in response to each one. This is a good way of dealing with the case when two or more referees suggest almost exactly the same thing. A list of your paraphrasings of the referees’ suggestions and a short statement about how you have responded to each one can make a very helpful executive summary that will reassure the reader that you have responded satisfactorily.

Some journals require you to do this but few funding agencies do. Even so, you should do it because it is likely to increase your chance of getting funded because it reassures the committee that you have responded satisfactorily to the referees’ criticisms.

Keep your overall response as short and simple as possible. This is more important with grant applications than with papers because the committee works under immense time pressure.

Know when to give up. If the referees’ reports on a grant are uniformly lukewarm it is unlikely to get funded whatever you say. Paradoxically, faint praise is more damning than strong condemnation because condemnation can give the impression that the referee is biased.

Of course, if you are overwhelmed by the desire to let the referees know how stupid their comments are, you probably won’t be able to follow any of the advice I have given so far. In that case I suggest that you wait until you can. Responding to referees is like inviting an  elderly relative to dinner, better not to do it than to do it with bad grace.

Recipe for a NIH Grant

ManhattanI’ve just spent a very pleasant few days in New York City teaching post-docs at  Mount Sinai my recipe for a NIH grant. We focused on the K99/R00 scheme, which is for applicants less than 4 years from their PhD. K99/R00 applications must  include career-development as well as  research, which makes them more complex than a pure research grant, but the recipe I developed can easily accommodate other grant schemes by adjusting  the formula for their assessment criteria. Before I describe the recipe – how you get the grant written – I need to say something about the formula – what the finished grant consists of.

The Formula

The formula is based on the principle that you should make it as easy as possible for those who carry out the peer review of your grant to do their job. NIH makes it very easy to apply this principle because they take great trouble both to explain their peer review process and to list the review criteria for every kind of grant they offer.

The peer review process is carried out by a study section, a group of about 15 scientists whose expertise covers the area of the grant. Three of them, the reviewers, have to produce a written assessment of the grant and score it against the review criteria before the study section meets. At the study section the reviewers present the grant orally, then it is discussed openly and then scored by secret ballot. All members of the study section join in the discussion and vote on the score even though they may not have read the grant, although they will probably have read the 1 page Specific Aims, or the 30 line Summary. So the formula has two requirements.

  • It must be very easy for the reviewers to understand the grant quickly and to find the detail that shows whether or not it meets the assessment criteria.
  • It must be easy for the other members of the study section to speed read it and understand it.

To make it easy for reviewers, the three versions of the grant – the 12-page Research Strategy, the 1 page Specific Aims and the 30 line Summary should be recognisably the same, so that someone who has read the Summary or the Specific Aims should find the Research Strategy looks familiar. I recommend that you define the grant with a set of about 10 key statements that address the review criteria. The key statements begin major sections of the Research Strategy and are followed by text that justifies them and convinces the reviewer that they are true. The reviewer will be familiar with the key statements because they appear in the same order in the Specific Aims and the Summary, but with less text to justify them.

The repetition of the key statements between the Summary, the Specific Aims and the Research Strategy helps the other members of the study section to feel that they understand the Research Strategy as soon as they glance at it. To make it possible for them to speed-read it, the top line of every paragraph should contain the main message of the paragraph and there should be white space between the paragraphs.

The Recipe

To cook up a grant with this formula you need to start by deciding what key statements you need. NIH assessment criteria are very helpful here.

  • All grants are required to have a number of specific aims – goals that the research hopes to achieve. Three is the ideal number of specific aims – just as it is the ideal number of points to make in an emphatic statement.
  • The Research Strategy document must be divided into three sections, each of which is assessed. They are Significance, how important the research is; Innovation, what is new about the research, and Approach, how you are going to do the research.

Each specific aim needs one key statement to state its significance and one to state the approach to achieving it. Innovation can be stated separately for each aim or in a single staement that covers all three aime. Thus you need between 7 and 9 key statements to cover the basic criteria.

In addition you should have a couple of key statements to introduce the Research Strategy. These make the ‘elevator pitch’. The first needs to say what overall outcome is hoped for, something about how it will be achieved and something about your credentials for carrying out the project. The second needs to say something about the project’s overall importance. You can think of these statements as the opening remarks of the reviewer addressing the study section. They correspond to key sentences 1 and 2.

Finally you probably need an ‘onwards and upwards’ statement to finish off the Research Strategy, something that says how you will take the research forwards at the end of the project.

The K99/R00 grant has a training requirement. The research project must contain a mentored (K99) component and an independent (R00) component. The candidate is also required to submit a training and career development plan. The criteria make it clear that the K99 project must develop the candidate’s skills to prepare them for the independent phase. One or two extra key statements are needed to address this aspect of the project.

Once you have drawn up your list of key statements you draft them. They are the introductory sentences of the main subections of the Research Strategy. Don’t spend too much time perfecting these sentences. Get on with writing the sections that they introduce. As you draft those sections you will naturally polish the key statements until they are as good as you can make them.

Once you have a complete draft of the Research Strategy you can copy the key statements from it into the one-page Specific Aims document. Make that document up to a page by cutting and pasting more text from the Research Strategy or by drafting new text. You should cut and paste wherever possible to maximise the overlap between the two documents.

There are a number of sources that help you with content.

These can help you to modify the formula by choosing different sets of key statements and by varying which key statements appear in the different components of the application. But the best way to minimise the pain of writing is to follow the recipe.

Say it again Sam. And use the same words.

groundhogRepetition of key sentences and key phrases is extremely important in a grant application. The key sentences that introduce each subsection of the background and the description of the project in the case for support should be repeated in the introduction and also in the summary.  So each key sentence should appear at least three times.

Some key phrases should be repeated more than three times because they occur in more than one key sentence. For example, imagine you are writing a grant in which one of the sub-projects will characterise the relationship between motherhood and apple pie.  The phrase ‘the relationship between motherhood and apple pie’ will be in two of your key sentences.  One will explain why we need to characterise ‘the relationship between motherhood and apple pie’.  The other will introduce  the description of the sub-project that characterises the relationship between motherhood and apple pie.

Most academics accept that it is helpful to repeat key sentences. But most of them reject the idea that the repetition should use  the same words in the same order.  So I want to explain now why it is more effective to use the same words in the same order whenever you repeat a phrase or sentence.

Effectiveness is much more important here than correctness.  Few would disagree with the assertion that exact repetition is a more correct use of English than paraphrasing but it is much more important to think about how you can increase the effectiveness of a grant application by using repetition in the way that I recommend and how you will fail to increase effectiveness in the same way if you change the words you use or their order.

In thinking about the effectiveness of a grant application, we should consider who will read it and how.  Committee members and referees have different needs and derive different benefits  from repetition.

The most important readers are the committee members that make the decision.  All of them will have a vote in deciding whether or not the grant application gets funded. Few, if any,  will understand the details of the research topic. All of them will read the summary and most of them will stop there. Some will try to read the application and understand it. Usually two  members of the committee, the designated members, are tasked with reading the application and leading the committee discussion. They will try hard to understand the application, but they will find it very difficult and they won’t have much time – maybe an hour. Any help you give them will be gratefully received.  Although most of the rest of the committee will not read the application they will probably glance through it during the discussion.

There are three ways that repetition is particularly helpful to committee members:-

  • Repeating the key sentences means that all the committee members will be likely to remember them. Even those who just glance through the application once will read the key sentences three times. This means that there is a very good chance that they will remember them and understand the logic of your case for support – what outcome your project will achieve, why it is important, what things you need to know in order to achieve the outcome and how you will achieve them. If you repeat the key sentences but substantially change the wording then people will be less likely to remember them. Every change in wording is likely to be interpreted as a change in meaning, leading to potential confusion.
  • Repeating key phrases in the sentences that state what we need to know and what the sub-projects will discover makes it very clear that the project will discover exactly what we need to know. In this way the key phrases act like labels for the different parts of the project.
  • Repeating the key phrases enables committee members to learn them and to have a sense of what they mean. Humans learn the meaning of new phrases by encountering them repeated in different contexts. Committee members who read your grant application carefully will get the sense that they know what it means , even if they don’t. If you vary the wording of the key phrases it becomes harder to learn them and  less clear to the reader that you mean the same thing.

The referees are, notionally at least,  experts in the research topic. They will read the application, write an analysis of its strengths and weaknesses and give it a score, which the committee will consider, but not necessarily follow.  The referees are likely to read your application more carefully than the committee members and to have a deeper understanding of the topic. However, they will want to assess whether the detailed content of your case for support actually supports the assertions made in the key sentences. Repetition helps them do this in two stages.

  • Reading the key sentences in the summary and in the introduction allows them to create a mental list of questions to which they want to find answers in the case for support.
  • Repeating the key sentences at the head of each subsection of the case for support guides the referees to the answers. Again, the key phrases act like labels. For example, a referee that has some doubts about whether your research approach really will characterise the relationship between motherhood and apple pie will be guided straight to the place where you describe the relevant part of your research approach by the phrase the relationship between motherhood and apple pie. If you decide to change any of the words in the key phrase, not only does it become less effective as a label, it also introduces the possibility that you are seeking to do a piece of research without having told the reader why it is important to do it.

In sum, repetition of key sentences and key phrases makes a grant application more effective in four different ways, provided that you use the same words in the same order.

 

 

We had the workshop: where are the grants?

16630702_sAt Parker Derrington Ltd we often encounter rather fixed ideas about how to improve grant-writing outcomes. ” A workshop is what we need….. Can you manage 100 participants? Can you tell them how to get bigger grants? Can you do it any cheaper? What can you do in half a day?”  Obviously, to survive in business we have to allow the customer to be right, so we find ourselves giving rather a lot of grant-writing workshops.

Of course we think our workshops are the best in the business and we get excellent feedback.  However, a workshop can only do so much. Even academics who have good ideas, a good track record of publications and who can design a fundable research project need more than a workshop can provide. Let me explain. There are three problems that tend to prevent such academics from writing good research-grant applications:-

  • They don’t appreciate the unwritten constraints on the case for support.
  • They don’t have an efficient way of writing a grant-application.
  • They don’t usually get high quality help and encouragement from other academics.

Let’s take these one by one.

The unwritten constraints on the case for support

There are two constraints that push people in opposite directions.

  • It must be speed-readable. Research councils don’t tell you that most of the committee members who take the decision on your research-grant application probably  haven’t read the case for support properly. Worse,  even if committee members did read the case for support properly, most of them wouldn’t understand it.  Obviously your case for support must enable speed-readers to understand and remember what you expect to discover, why it is important, how you will do it, and what you will do with the results.
  • It must be easy to find the detail. Your case for support will also be read by expert referees who will want to assess the detail of what you will do and why it is worth doing. However, few people appreciate that referees will do a much better job and will feel happier about doing it if you make the job easy. So your case for support should  guide referees to the specific content that supports each element of the case. And your summary should make it absolutely clear what arguments the case is making, so that referees know before they begin reading the case for support, what arguments they want to test.

In our workshops we explain how these constraints arise and how to design a case for support that meets them.

An efficient grant-writing process

It should be possible to write a research-grant-application in a week. Most people take months. Some take years.

We find two common factors that make writing inefficient. Probably the commonest is starting to write the grant proposal before designing the research project. Remember, the grant application is a marketing document that is trying to secure investment in the project. It can take a very long time to write it if  you start before you define the project because you don’t know what you are marketing.  Moreover, applications that are not based on a defined project usually fail to convince the reader that they are marketing anything at all: we call them zombie grants. A second factor that makes writing inefficient is not having a guide that tells you what to write in each part of the case for support.

We teach a 2-stage approach to writing in which the first stage is to write a summary that consists of 10 key sentences. That summary is enough to check whether the writer has a viable project. In the second stage, the summary is a guide that tells you what to write in each part of the case for support. Each of the key sentences  forms the first sentence of a major sub-section of the case for support and defines what the rest of the sub-section must convince the reader to be true, either with evidence or with detail about proposed research activities.

How to help a grant-writer.

It’s harder to help a grant-writer than you might think. It should be easy to give clear feedback that will tell them exactly what they have done wrong. Actually, it’s quite hard to do that unless your heart is made of stone, because accurate feedback is crushingly demotivating. Telling a colleague what is wrong with a grant-application that has taken them six months to write can completely destroy their motivation.

To write a grant quickly and well, a writer needs encouragement and feedback, delivered as directly and as quickly as possible.  Our 2-stage approach to writing makes it easier to give constructive feedback, partly because it breaks the writing task down into smaller chunks but also because it makes it easier to define what is expected at each stage, so it combines feedforward with feedback. We offer workshops for coaches to help academics coach their colleagues it but we also offer coaching directly to writers, either as a stand-alone service or as a follow-up to a workshop.  If you are interested, get in touch.

First you tell them; then you convince them.

Quotative Like; xkcd.com

Some  common writing styles are very bad for grant applications and this post aims to help you to avoid one of the worst.  It is a style of writing that we refer to in the book as “Argue – conclude”.

Argue-conclude writing sets out the argument for a statement before it makes the statement.  Done well, argue-conclude writing can be very convincing for a dedicated reader,  who will follow every twist and turn of  your argument. By the time they get to read a statement that ordinarily they might be inclined to reject, they already know the arguments that support it. Unfortunately, most of the readers who will decide whether your grant application gets funded are less dedicated. They will give up reading before they get to the crucial statement.

To communicate with these readers, you begin each paragraph with its main message. Then use the rest of the paragraph to convince them that the message is true. In the book we refer to this style as “assert-justify“. An easy way to describe about assert-justify style is “Tell them; then convince them”.

As I was writing this I thought of nine reasons you should adopt “Assert-justify” style in research grant applications.  The first four are concerned with meeting the needs of the reader – one of the guiding principles for writing with style. The remaining five are concerned with making the task of writing easier. Naturally I shall assert each reason and then justify it.

  1. Assert-justify style communicates more effectively with speed-readers, tired readers, and lazy readers.
    These readers will skim through your document. The neurology of eye-movements dictates that, provided you put blank lines between the paragraphs, their eyes will skip from paragraph to paragraph. They will read the first line of each paragraph. Thus they will read the assertions and get the headline messages. If they are inclined to disagree with the headline messages, they will dig down into the arguments that justify them.
  2. Assert-justify style makes it easier for diligent readers, such as referees, to examine your arguments in detail.
    Each paragraph starts by stating what the paragraph is about. This makes it very easy for the reader to find the arguments they want to examine. They never face the problem of wading through an argument wondering where it is leading.
  3. Assert-justify style makes it easier for the committee-member who has to present your grant to the rest of the committee.
    They can see at a glance what points you are trying to make. This makes it very easy for them to select the points that are most important and relevant for the committee, even if they don’t entirely understand them.
  4. Assert-justify style is more likely to engage readers who are bored.
    The conclusion is always the most interesting part of the argument. By putting the conclusion first you are more likely to entice them to read.
  5. Assert-justify style makes it easier to write an accurate summary.
    The assertions from each paragraph comprise a draft summary. If you want a shorter summary you may be able to leave some of them out.
  6. Assert-justify style makes it easier to write a good introduction.
    The assertions from each paragraph comprise the core of the introduction. You may need to add some linking text and some signposts.
  7. Assert-justify style makes it easier to write short sentences.
    You can write in simple, clear statements. You don’t need to frame them and qualify them.
  8. Assert-justify style makes it easier to write short paragraphs.
    In argue-conclude writing you have to spend a lot of words preparing the ground for the argument. If you start by asserting the point you want to make, you leap straight into the argument without spending any words.
  9. Assert-justify style makes it easier to write.
    I used to spend a lot of time staring at my screen wondering how to get started on each section. In assert-justify writing you can write the ten key sentences that start each sub-section of a grant proposal in an hour.

There are probably more and better reasons to write in assert-justify style. When I started writing this post a couple of hours ago I only had three!  If you have any doubts about whether assert-justify style is correct, it may help you to read Andy Gillett’s recommendations  on the Use of English for Academic Purposes blog.

Let me finish with an example of what I think you should avoid. This abstract of a funded grant application is short and clearly written but it is in argue-conclude style; consequently the piece of information that the reader most wants to know – what will the research project do – is buried away in the second half of a sentence in the last paragraph. A speed-reader would not see it.

Committees and Referees

Committees Like a Simple Story: thanks to Science and Ink http://www.lab-initio.com

Committees prefer a clear story: http://labinitio.com

The Journal Nature reported yesterday that scientists have complained that there is a mismatch between expert referees’ evaluation scores of research grant applications and funding decisions.  Different interviewees claimed that this mismatch either does or does not indicate either a flaw in the system or mistakes by referees or by committees.

There may be flaws in the system and mistakes probably happen, but there is a more obvious reason that referees’ scores should not be expected to predict funding decisions. Referees and committees do different jobs which impose different constraints on the way a grant application is written. Very few proposals are written in a way that satisfies both sets of constraints, and so, for the majority of proposals, there is no reason to expect a close match between the referees’ score and the committee’s score. Before I explain the constraints and how to meet them, I’ll clarify the story and explain its relevance.

The story was based on 302 grant applications to the Medical Research Council. It states that some applications that received high scores from the referees were triaged (rejected without being discussed by the committee). Of the proposals discussed by the committee, the story states that the group of applications that were successful and the group that were rejected had ‘a nearly identical spread of scores’. It’s worth noting that the story focuses on a statistic of the scores that is not particularly informative (spread) and does not mention any other statistics. It is relatively common that the referees include both friends and foes of the applicant, which can cause the spread of scores on a single application to range from the lowest to the highest possible. Consequently, nothing that the story says about this rather small data sample indicates a general failure of referees’ scores to predict committee decisions.

However, whatever we may try to conclude from this small dataset, most funding agencies (the EPSRC is an exception in the UK) ask referees and committees to do very different jobs. These jobs depend on different aspects of the way the proposal is written.

Referees work alone and each one works on a single application. The referees are expected to be experts in the same research field as the applicant and their evaluations are typically seen as coming from within that field. Their main task is to test the detailed rationale of the proposal. Are the research questions important? Is the research approach feasible? Does the research team have the ability to carry out the project? What is their standing in the field? Has their previous work made a significant contribution? Is the approach novel? Are other teams likely to get the answers first?

To get a good score from a referee, a grant application must contain relevant detail. Evidence that the questions are important and relevant must be cited. Any evidence to the contrary must be dealt with effectively. The research approach must be described clearly and in sufficient detail to convince a knowledgeable sceptic that the project is feasible, can be carried out with the resources requested, and will lead to the promised outcomes.

The committee works as a collective and takes a view across all the applications before them, typically about 100 for a single meeting. They must also take a view across all the different research fields that the committee supports. They need it to be clear that the project will deliver an  outcome that will have importance beyond the immediate research area and that the applicant has identified an approach that is likely to be productive and that the research team has the skills to deliver it. They need to know what the research aims are and how they relate to the overall outcome. They need it to be clear that the research objectives will satisfy the research aims and deliver the overall outcome and that the results will have appropriate impact

More importantly, the committee also works very quickly. They have to reach an agreed view about grant applications which most or all of them may not understand completely. For the committee to score an application highly they need it to be possible to understand it on the basis of a single hasty reading – or even a quick skim. The case for support should have a very clear structure that states clearly what will be the research outcome, why it is important, what are the research aims, what are the objectives and what will be done with the results.

Of course the best possible case for support has a clear structure that enables the committee to understand it and appreciate its strong points. It also uses the structure to guide the referees to the relevant detail. In my experience cases of support written in this way get very high scores both from referees and from committees. Unfortunately they are very rare but this blog explains in several different ways how to write them.

 

Build the Project, Then Fit the Question

Was your question clearly linked to your project?

Is your question important to the funder?

The first thing you must do in a grant application is convince the reader that you are going to address a question that is important to the funder. In chapter  7 of The Research Funding Toolkit, we refer to this as the ‘Importance Proposition‘.

The importance proposition is fundamental: your grant will definitely not get funded unless you convince the reader that your research question is important to the funder. Even so,  you should take a project first approach. Generate your project and then fit the question to it. Do not try to pick an important question and then design a project to solve it. The project first approach is quicker – often by several months – and it reduces the risk of writing a zombie grant.

The project first approach is easier if you take a modular approach to project design and start by generating a catalogue of sub-projects.  As you generate each sub-project, you should ask  yourself if the outcome will contribute to answering an important question. If it will, keep the sub-project and begin preparing yourself to make that case. Otherwise, discard that sub-project and try  to generate another one with an outcome that will contribute to answering an important question.

Once you have a few sub-projects – ideally at least five or six – it is easy to generate projects. The ideal project has three, or just possibly four, sub-projects.  When you combine sub-projects to generate a project you need to start looking hard for a good research question. You need a question that covers all  of the sub-project outcomes comfortably. It is better to take a question that is too big to answer, and answer it partially, rather than risk picking a question that is too small to be exciting and answer it completely.

An important part of your development as a researcher is to develop the ability to design projects that produce results that help to answer important questions. I absorbed this from the culture of the lab in which I did my PhD and this is part of the approach we recommend in the book. However, it is also possible to search on the web to see if a given funder will fund the kinds of research outcomes you are likely to produce.  Obviously every funder’s website will have a statement of their remit, but this can be hard to interpret because it will be couched in terms of questions rather than outcomes. A better way to get a sense of the outcomes that excite a given funder is to scan their press releases. Best of all, some funders have a database that includes the abstracts of all their funded projects.

The Gateway to Funded Research is a searchable database that covers  all the UK research councils. The Projects and Results page on the European Research Council website is also searchable and allows you to see research outcomes.