Tag Archives: Referees

The Magic Formula for a Case for Support

I want to explain my solution to the readership problem I described in my last post. The solution is a little bit complicated and at different times I have described it in different ways, trying to come up with fancy acronyms like PIPPIN. However, we can just think of it as a magic formula for a case for support.

The way the magic formula works is that it creates the case for support as a series of layers. Each layer takes care of some of the needs of the very different groups of readers that matter. The readers that matter are those who participate in the funding decision.

Who are the Readers that Matter?

There are three groups of readers who participate in a funding decision, ordinary committee members, presenting members and referees.

Ordinary Committee Members

Most of the votes that determine whether your application is funded come from ordinary committee members who have really deep expertise in areas completely outside the topic of your project. Unfortunately, when it comes to your topic, they probably don’t understand it. They may not even see why anyone would want to research it. You need to convince the ordinary committee members that your project is worth some of the money that might otherwise go to projects on topics that they know are important.

You don’t have much opportunity to convince them because your application is one of a very large bundle that they don’t really have to read. They only need to know enough about it to get through a discussion, in which they can remain silent, and vote on a score, for which there will be a recommendation. They will be much more concerned to spend time reading the applications that they have to present to the committee.

However, you do have a chance. They will want to decide whether to push your score up or down, which they can only do by understanding your application. However, bitter experience tells them that, for most grant applications, understanding does not come easily, if at all. They will want to check if yours is worth the effort. They will probably have a quick look at the introduction before the meeting. You have maybe 30 seconds to catch their interest.

If you do manage to catch their interest by persuading them that the topic of your research project might be important enough for them to want to see it funded, and that your application might be intelligible, they will dig a little bit deeper. They might spend five minutes skimming through your application trying to work out whether your project will make enough progress that it should be funded. They won’t understand your technical terminology. You have write in such a way that they can work out what they need to know from the context.

Presenting Members

A couple of committee members will know a bit more about your topic, and spend a bit longer reading your application. They may even think your topic is important, but they probably won’t be real experts. They will have the job of presenting your application to the rest of the committee, and recommending a score. They won’t have a huge amount of time to read the application because they probably have eight or nine others to present on the same day. You will probably have about an hour to make them expert enough to explain why the problem you are trying to solve is important and to master the detailed evidence that your project is likely to make good progress towards a solution.

Even if the presenting members are enthusiastic about your project, the score they recommend is likely to be quite conservative. A conservative recommendation is inevitable if the presenting members can’t understand the application; they will just follow the referees’ recommendations without much enthusiasm. However, even if they do understand an application, presenting members will be aware that they are not expert enough to know whether your work really leads your field, which means they have to be conservative in their recommendation. Their colleagues on the committee will be aware of their limitations, and so will be sceptical of over-enthusiastic recommendations. In a world of conservative recommendations, the enthusiasm of committee members can make the difference between success and failure.

Expert Referees

Referees are real experts on your topic. They want to see detailed evidence that your topic is important and that the research questions your project addresses are important and that your project is likely to provide answers. They will then make an expert judgement and write a report and recommend a score that accompanies your application when it goes to the committee. They can probably spend several hours reading your application because, unlike the committee members, who get a big bundle of applications at once, referees get sent applications one at a time.

The Layers of a Case for Support

These three groups have very different needs. Any group can kill an application, so you have to write something that appeals to all the groups. It also helps if what you write gives them a common language to discuss the case for support. The magic formula constructs the application so that it consists of multiple layers that work together, to meet the needs of all the groups of readers without alienating any of them.

Top Layer: The Opening Sentence

The opening sentence has three tasks.

  1. It has to catch the interest of the ordinary committee member and make them feel that it might be worth reading a bit of your application even though they don’t know anything about the topic. I always begin the sentence with a ‘big picture’ statement about the goal of the project. It is important to express the statement in language that will be intelligible to ordinary committee members.
  2. It has to speak to the expert in a way that will reassure them that your project has some real ‘meat’ to it despite the rather bland and aspirational ‘big picture’ opening. I try to make sure that the sentence continues with a much more specific statement that builds trust in the methodological rigour of the project and the competence of the research team. Ordinary committee members probably won’t understand the detail here but if the opening statement is good enough they will accept that this is just a more specific version of it.
  3. The sentence must serve as a summary of the case for support, in case the reader chooses not to read on.

Second Layer: The Introductory Summary of your Case

The second layer of the case for support is a summary of your argument that you should be funded. The magic formula provides the argument in ten sentences, including the first sentence. These sentences are the first ten sentences of the case for support. I have already talked about the opening sentence. The remaining nine sentences are:-

  • A sentence stating what makes the topic of the project important to the funder.
  • Three sentences, each one stating an important research aim relevant to the topic, and a reason that research aim is important. The research aims might be couched in terms of hypotheses to be tested, relationships to be established, questions to be answered or in some other way. But in every case there will be something that could be achieved by an as-yet undefined research project, and a reason that it is important to achieve it. I call these sentences the problem sentences.
  • A sentence giving an overall description of the project.
  • Three sentences, each one describing the research in a part of the project, and what that research will achieve. The part of the project might be referred to as an objective, a work package, or in some other way. What the research will achieve will be expressed in the exactly same words as were used to express the research aim in the corresponding problem sentence. Using the same words makes it clear to every reader that all of the important research aims referred to in the problem sentences, will be achieved by the project, , whether or not they understand the terminology used to describe the research aims and outcomes.
  • A final sentence saying what will happen as the project concludes.

This second layer provides all the readers with exactly what they need: a short simple statement of your argument that you should be funded. The essence of that argument is that your project deals with an important topic, and will achieve important research aims related to that topic. By expressing the research outcomes in exactly the same words you used to express the important research aims, the magic formula ensures that all the readers will understand that the project achieves those important aims, whether or not they understand the terminology used to express them.

This statement of your argument is probably as much as the ordinary committee members have time for, which is why you put it right at the beginning. You would now like to ensure that all the readers believe your argument and remember it. To do this you recycle the sentences from the introduction to create a framework for the detailed evidence.

Third Layer, Structure of Case for Support

The third layer of the case for support is provided by its structure. After the introduction there are nine more sections in the case for support. Each of those sections begins with one of the sentences from the introduction and contains all the evidence that you will draw on to convince the reader that the sentence is true. The section will also have a heading that is a shortened version of the first sentence.

Creating the framework from the same sentences as the introduction has the effect that, just by skimming the case for support, a reader will see that it marshals evidence to support every part of the argument in the introduction. If they already believe the argument, they probably feel no need to read on, which is probably the case if they are just an ordinary committee member. On the other hand, if they want to test all or part of the argument, they can see exactly where to find the evidence.

Fourth and Fifth Layers: Internal Structure of Sections and Paragraphs

The referees and the presenting members want to know what evidence you are using to support each part of your argument. I recommend that you begin each section by stating the main points of the evidence. Then you make each point of evidence in a single paragraph. I also recommend that you begin each paragraph by stating the point you want to make. Then you use the rest of the paragraph to make the point.

The advantage of this hierarchical structure is that readers can find the detail they need if they want to test any of your points, which the referees probably will. However, readers who don’t want to test the detail, which the presenting members probably won’t, can learn what it is and then skip over it by looking just at the beginnings of the sections and the tops of the paragraphs.

The structure of the sections and paragraphs ensures that any reader can check the evidence you are using to support your case, at any level of detail. The fact that you make repeated use of the sentences you used to summarise your argument in the introduction helps readers to find their way through the evidence, and, incidentally, helps to make sure that they remember your argument and that they all express the argument in the same terms.

Sixth Layer: Summaries

Not all members of the committee will have time to read your case for support. To ensure that committee members who don’t read the case for support know its argument, I recycle the sentences of the introduction in any abstract, summary or statement of aims and objectives that forms part of the application form.

The Readership Problem.

Most of the research grant applications that I read make me think that the writer is trying to solve the wrong problem. I’m referring specifically to the problem they are trying to solve by writing the case for support. The most important problem to solve is the readership problem – how to convince the readers who participate in the funding decision that your research topic is important and that your project will make good progress.

There are two ways that writers try and solve the wrong problem.

  1. In the worst case, writers use the writing process to try and solve the problem of what to write. What is the story they want to tell in the case for support? This is a problem that they should have solved before beginning. Trying to work out the story of the case for support by writing it produces a hopeless mess.
  2. Many writers work out the story of the case for support before they start writing but they design their case for support to communicate with the wrong audience in the wrong way. This approach usually produces an application that is fundable in principle – indeed the vast majority of funded applications are like this – but which would have a better chance of funding if it were designed for the readers who will make the funding decision.

Some writers, particularly early career researchers, use the writing process to refine their research topic and then to work out the design of a project that addresses the topic. They are hoping that the writing process will enable them to create a well-defined and well-justified project. It never does. There are two reasons:-

  1. Refining the research topic has no natural end point and most people continue until they have raised more questions than could conceivably be answered by a single project. Worse, the detailed development of the research topic takes so much space that they don’t have enough space left to describe a project.
  2. A detailed set of research questions is a bad starting point for the design of a project because it is likely to be impossible to design a project that answers them. Remember, the research questions in a grant application are really just a sales device, so it is much better to design the project first and then work out what research questions will create the best sales pitch.

Other writers, including most experienced researchers, start with a well designed project and try to solve the problem of cramming as much technical detail as possible into the case for support. They reason that the expert referees who read the case for support will penalise them if they omit any relevant details. They compound matters by seeking advice from close colleagues who suggest further details. The case for support becomes such a mass of detail that they need to reduce page margins and font size and devise ingenious abbreviations so that the mass of detail fits within the page limit.

Cramming in detail does not produce an effective case for support. To be effective, the case for support must make a clear argument that the research topic is important and the project is likely to be successful. And that argument must be accessible to every person that participates in the funding decision. Of course detail is necessary to convince referees of the merits of the argument, but detail on its own is not enough, and even if it were, the referees do not make the decision. They make a recommendation that accompanies the application to the grants committee, which decides whether or not to fund it. A negative recommendation from the referees is likely to kill an application, but a positive recommendation does not guarantee success. An application that fails to communicate its argument to the committee – most of whom do not have time to read the detail and would not understand it if they did – is likely to fail.

To be most effective, the case for support must convince readers with different levels of understanding, who will read it in different ways, that the research topic is important and the project is likely to make good progress. There are three groups of readers who participate in the decision:-

  1. Referees are experts on the topic and will analyse the case for support in detail.
  2. A small number of committee members will have a fairly good understanding of the topic and will read the case for support carefully.
  3. Most of the committee will have only a hazy understanding of the topic and will spend very little time reading the case for support.

Designing a convincing argument, and then setting it out in a way that works for all three types of reader, is the most important problem to solve in writing a case for support. Many academics think the problem cannot be solved, except possibly by a magic formula. In my next post I will explain how my approach to writing the case for support solves the problem.

What’s the Point?

I want to explain why I think it’s better to produce a well-written grant application than a poorly written one. Obviously, given the nature of my business, I have to make this case, but it is not as simple as you might think, not least because most successful grant applications are very poorly written. In fact, if you think carefully about the quality of grant applications, it becomes clear that, in this particular domain, quality is completely subjective. So I will start by saying what I think makes a good grant application.

The quality of a grant application is not the same as the quality of the research project it describes. A grant application is essentially a marketing document for a research project and you can have a first-rate application that markets a tenth-rate project. And vice versa. Indeed  poorly-written grant applications are very often successful precisely because grants committees are trying to judge the quality of the project, not the quality of the application. Judging the quality of a project can be very difficult if the application is poorly written. So what makes a good grant application?

The essence of a good grant application is that it makes it easy to judge the project. The application contains all the detail that an expert will look for. The detail should be set out so that it can be read at very high speed and understood by a non-expert. As a rule of thumb, it should take less than two minutes to understand the main points of what you will do and why it is worth doing.

Those main points should be expressed and justified in such a way that a non-expert ‘gets’ what you are going to do and why. An expert should also be able to drill down and find the detail that they need in order to judge whether your project is likely to succeed and achieve those main points. I have already explained how the ‘key sentence’ structure enables a grant application to fulfil these requirements.

Despite the fact that most successful grant applications are poorly written, there are three reasons that it is worth taking trouble to produce a well-written grant application:-

  1. If your project is good, a well-written application will improve your chance of success.
  2. If your project is bad, a well-written application will help you to see that it needs to be improved.
  3. A well-written application can be easier and quicker to write than a badly-written application.

I’ll deal with the first two reasons in this post and I will deal with the third in another post.

Well written applications are more likely to be successful.

Well-written applications generate an enthusiasm among committee members that makes them give higher scores. For reasons I’ll explain in a future post, the person leading the discussion is likely to recommend a relatively conservative score, no matter how much they like the application. But if the committee are enthusiastic, they are quite likely to argue that the recommendation should be raised, and to exceed the recommendation when they score.

Poorly written applications can also get high scores, particularly if the referees have given very strong recommendations, but when committee members don’t understand an application they will not argue for a higher score and they may even score slightly below the recommendation. The consequence is that the scores of poorly written applications tend to drift downwards. The effect is small, but if the score is close to the borderline, which is likely to be the case, given the tendency for conservative recommendations, a tiny drift can make the difference between success and failure.

A well written application helps you see that you need to improve your project.

A well-written application explains your project very clearly at two levels.

  • First it explains what makes the project important to the funder.
  • Then it explains what the project consists of, and why each part of the project is important.

If your project needs to be improved, you are likely to find one or both of these explanations unconvincing as you write them. If you do find yourself writing arguments that you find unconvincing, then you need to reexamine your project and work out how to make it more convincing. If your application does not convince you, it is unlikely to convince a committee.

We thank the referees for their helpful suggestions…….

PeerReviewCartoonIt may not be easy to keep your temper when responding to referees’ comments but you must make it seem like the easiest thing in the world. Whether the comments are on a grant application or a paper, you should compose your response with regard to the effect you want it to have on the reader. If you want your response to influence a grants committee to award your grant or a journal editor to publish your paper, anger is unlikely to help.

There are some differences between responding to grant-application referees and paper referees but most of the following recommendations apply to both.

Assume that the referee is trying to be helpful. You will do a better job if you can make the assumption of reasonableness. If you can convince yourself that the referees are doing their best to help you  and their comments are genuinely trying to help you improve your research project or your paper, it will be easier to follow most of the recommendations below. It doesn’t matter if the assumption is wrong and the referees are actually trying to sink your grant or turn your paper into tedious gibberish. The important point is that you will do a better job of responding to them if you assume that they are trying to help.

Take responsibility for the reader’s failure to understand. Some writers take pride in the knowledge that very few people understand their work. I have seen responses from grant applicants that berated referees for their stupidity. That is a risky attitude to take in any type of writing and is almost certain to lead to the failure of a grant application.

Express gratitude for referees’ suggestions. Derrington’s first law of responding to referees states “The less you feel gratitude, the more fulsomely you should express it”. This is probably more important in the case of a paper than a grant application because in the case of a paper the referee will usually be consulted about whether your response is satisfactory whereas in the case of a grant application your response will be usually assessed by the grants committee.  Of course it may help you to express gratitude if you can explain how the referees insightful suggestions have enabled you to transform the research project or paper.

Make it clear that the responding to the referees’ suggestions has enabled you to improve the research project (or paper). There are two reasons for this.

  • First, you want to give the impression that the final version of your paper (or grant application) is much better than the version that the referee evaluated and deserves a higher score than the referees gave it.
  • Second, you may want to distract the referee (or the grants committee) from the fact that you may not have made any changes at all. In this case, Derrington’s second law of responding to referees applies “The less you change, the more emphatically you state how much you have changed and how much this has transformed the paper.”

Derrington’s second law is more important for papers than for grants, because journal editors often regard referees’ recommendations as binding and authors often find them unacceptable. A grants’ committee is much more likely to recognise when a referee’s recommendation is ridiculous and accept a response that politely declines to implement it.

State clearly what you are responding to and how. This is particularly important when referees’ comments and suggestions are vague or ambiguous. It is often helpful to paraphrase the referees’ suggestions and to state what changes you have made in response to each one. This is a good way of dealing with the case when two or more referees suggest almost exactly the same thing. A list of your paraphrasings of the referees’ suggestions and a short statement about how you have responded to each one can make a very helpful executive summary that will reassure the reader that you have responded satisfactorily.

Some journals require you to do this but few funding agencies do. Even so, you should do it because it is likely to increase your chance of getting funded because it reassures the committee that you have responded satisfactorily to the referees’ criticisms.

Keep your overall response as short and simple as possible. This is more important with grant applications than with papers because the committee works under immense time pressure.

Know when to give up. If the referees’ reports on a grant are uniformly lukewarm it is unlikely to get funded whatever you say. Paradoxically, faint praise is more damning than strong condemnation because condemnation can give the impression that the referee is biased.

Of course, if you are overwhelmed by the desire to let the referees know how stupid their comments are, you probably won’t be able to follow any of the advice I have given so far. In that case I suggest that you wait until you can. Responding to referees is like inviting an  elderly relative to dinner, better not to do it than to do it with bad grace.

Committees and Referees

Committees Like a Simple Story: thanks to Science and Ink http://www.lab-initio.com

Committees prefer a clear story: http://labinitio.com

The Journal Nature reported yesterday that scientists have complained that there is a mismatch between expert referees’ evaluation scores of research grant applications and funding decisions.  Different interviewees claimed that this mismatch either does or does not indicate either a flaw in the system or mistakes by referees or by committees.

There may be flaws in the system and mistakes probably happen, but there is a more obvious reason that referees’ scores should not be expected to predict funding decisions. Referees and committees do different jobs which impose different constraints on the way a grant application is written. Very few proposals are written in a way that satisfies both sets of constraints, and so, for the majority of proposals, there is no reason to expect a close match between the referees’ score and the committee’s score. Before I explain the constraints and how to meet them, I’ll clarify the story and explain its relevance.

The story was based on 302 grant applications to the Medical Research Council. It states that some applications that received high scores from the referees were triaged (rejected without being discussed by the committee). Of the proposals discussed by the committee, the story states that the group of applications that were successful and the group that were rejected had ‘a nearly identical spread of scores’. It’s worth noting that the story focuses on a statistic of the scores that is not particularly informative (spread) and does not mention any other statistics. It is relatively common that the referees include both friends and foes of the applicant, which can cause the spread of scores on a single application to range from the lowest to the highest possible. Consequently, nothing that the story says about this rather small data sample indicates a general failure of referees’ scores to predict committee decisions.

However, whatever we may try to conclude from this small dataset, most funding agencies (the EPSRC is an exception in the UK) ask referees and committees to do very different jobs. These jobs depend on different aspects of the way the proposal is written.

Referees work alone and each one works on a single application. The referees are expected to be experts in the same research field as the applicant and their evaluations are typically seen as coming from within that field. Their main task is to test the detailed rationale of the proposal. Are the research questions important? Is the research approach feasible? Does the research team have the ability to carry out the project? What is their standing in the field? Has their previous work made a significant contribution? Is the approach novel? Are other teams likely to get the answers first?

To get a good score from a referee, a grant application must contain relevant detail. Evidence that the questions are important and relevant must be cited. Any evidence to the contrary must be dealt with effectively. The research approach must be described clearly and in sufficient detail to convince a knowledgeable sceptic that the project is feasible, can be carried out with the resources requested, and will lead to the promised outcomes.

The committee works as a collective and takes a view across all the applications before them, typically about 100 for a single meeting. They must also take a view across all the different research fields that the committee supports. They need it to be clear that the project will deliver an  outcome that will have importance beyond the immediate research area and that the applicant has identified an approach that is likely to be productive and that the research team has the skills to deliver it. They need to know what the research aims are and how they relate to the overall outcome. They need it to be clear that the research objectives will satisfy the research aims and deliver the overall outcome and that the results will have appropriate impact

More importantly, the committee also works very quickly. They have to reach an agreed view about grant applications which most or all of them may not understand completely. For the committee to score an application highly they need it to be possible to understand it on the basis of a single hasty reading – or even a quick skim. The case for support should have a very clear structure that states clearly what will be the research outcome, why it is important, what are the research aims, what are the objectives and what will be done with the results.

Of course the best possible case for support has a clear structure that enables the committee to understand it and appreciate its strong points. It also uses the structure to guide the referees to the relevant detail. In my experience cases of support written in this way get very high scores both from referees and from committees. Unfortunately they are very rare but this blog explains in several different ways how to write them.