Tag Archives: Research Grant Application

Scrivener for Grant Writing

Scrivener is designed for assembling a long document from a large number of components.

Scrivener is designed for assembling a long document from a large number of components.

This post is about using Scrivener to write a grant application. I have wanted to write the post for a long time but I was held back by the fact that it is about an idea that, until last week-end, hadn’t come to fruition. The idea is that Scrivener is the ideal tool for writing a grant application.

Scrivener’s strength is that it is designed for the task of assembling short sections of text into a larger document and for editing and rearranging the segments until they work together. This is exactly the problem with a grant application. You have to assemble the background information about your research problem and the description of your research project and then edit and rearrange them until they work together as a coherent sales pitch for the project.

For the sales pitch to work, reading the background information about the research problem must make the reader want that research project and no other.  This requires the content of the background to be exactly tuned to the content of the project. This kind of fine tuning is much easier if you draft the corresponding subsections of the background and the description of the project,  which will end up several pages apart in the final document, side by side.

Scrivener was designed to do exactly this.  It is a program designed for writing novels. I discovered it through Twitter. Someone I follow, whose writing I admire, said they found it invaluable, so I wanted to find it invaluable too. But the sad truth is that I never have.  My problem is that Scrivener isn’t very easy to use. In fact it’s pretty difficult. Everything I have ever tried to do with Scrivener is much easier to do with a different program. Last week end, this changed.

Last week-end I had the grant-writing assignment from hell. I had four days to assemble a case for support. The raw material I had was patchy, to say the least. I had about 30 pages of text provided by eight academics scattered across four continents. 95% of the text was about research but the grant was about building research capacity. Important components of the research project were still being written, whereas the source of text on building capacity and capability had run dry. The least of my problems was that the grant was a kind I had never written before, requiring the case for support to have a unique structure and a limit of 10 pages.

The first thing I had to do – fortunately I had done this the week-end before – was to overcome Scrivener’s two great weaknesses.

  • It isn’t very good at producing Microsoft Word, which is pretty much the only medium in which the academics I work with can collaborate on a text.
  • Its approach to references and bibliographies is extremely limited.

I needed to find a way that I could convert Scrivener text into Microsoft Office, completely reliably and with very little effort. I needed to get Scrivener to interface with a good referencing software package so that I could combine the citations provided by all the different contributors into a single bibliography, in a compact format.

Fortunately I had discovered an excellent blog post  by David Smith explaining how to use a program called pandoc together with the excellent reference software Zotero to produce Microsoft word output from Scrivener. Pandoc converts a text file produced by Scrivener into a Word document, which uses styles defined in a document of your choice, and Zotero inserts and updates and formats citations and a bibliography, which has a style determined by a csl file, thousands of which are available on Zotero’s website.  I had never used Pandoc or Zotero, so it took me about a day and a half to become proficient enough with them to get the process working reliably. While I was getting pandoc to work I also found this csl file, which makes pandoc produce bibliograpies in the style of an obscure Italian law journal that does not require the titles of journal papers.

I was slightly apprehensive that Pandoc, which is a 1970’s style unix command-line application, requires you to type a monstrously long command into a terminal app. This is the command I used.

$ pandoc --filter pandoc-citeproc --reference-docx=/Users/amd/Dropbox/02Andrew/11-Scrivener/pandoc/reference.docx -s -S --normalize  -f markdown -t docx -o CaseForSupport.docx -i CaseForSupport

This command specifies the citation process, the path to the Word document with the correct styles, the format and the name of the input file,  that the output should be  a standalone file, that repeated white space should be stripped out of the input, the format to convert to and the name of the output file and the name of the input file. Other details, such as the datafile and format specification for the bibliography were specified in the input file but they could also have been specified in the command.  It takes a few tries to get the command right but once you have got it right you can repeat it whenever you need to by pressing the up-arrow  key.

Once I had taken care of its weaknesses, I could start exploiting Scrivener’s great strength, its facilities for assembling a large text document from a series of snippets of text that have to be organised in different ways. Text came to me as Word documents, which I converted to raw text, marked up with formatting instructions in ‘Pandoc-flavoured Multimarkdown’. These instructions are very simple, and they cover the bare minimum – six levels of heading, numbered and bullet lists, bold and italic –  so it only takes a few minutes to mark up a whole case for support and it is very easy to keep the mark-up consistent. It reminded me of writing my thesis in the 1970s, using runoff on a minicomputer, which was a machine the size of a wardrobe with 0.05 Mb of memory, 0.25 Mb of disk that cost £25,000 (£140,000 in today’s money).

Exchanging edits and comments with colleagues was easy. I would send them the draft case for support as a word file and they would send back the same file edited with track changes, send me comments, or send me more text. I had to convert their edited text back to raw text, mark it up, and put it into Scrivener. This was quick and easy because I kept each contribution as a separate file, and it avoided the formatting errors that frequently happen when you merge Microsoft Word files.

Adding references was very easy. As long as the writer could identify the paper I would be able to find it and add it to my Zotero database. This was easiest if the writer used the DOI or the Pubmed ID but in several cases I  succeeded with nothing more than a google search for the author names, year, and journal title.

The funder’s instructions for writing the case for support required seven different sections so I created a folder for each section and sub-divided the contributions from different writers between the sections. Scrivener makes it very easy to split a file at any point. You can also select text in the file to use as the filename. This makes it easy to create separate files for each little subsection and to give them names that tell you what is in them. By the time I finished editing the case for support it comprised about 30 files.

Scrivener's Corkboard makes it easy to rearrange the sections of a document.

Scrivener’s Corkboard makes it easy to rearrange the sections of a document.

Using separate files makes it very easy to reorganise the text if you decide the structure is wrong, as I did reluctantly when one of the contributors pointed out that the case for support looked as if the main point of the grant was research, whereas the funding call was for building research capacity and capability. Completely restructuring the case for support involved splitting one file into six or seven pieces to be distributed throughout the document and reorganising the other files so that I now had a two level structure, with folders and files. Scrivener has a very flexible view of folders and files, you can convert one into the other, but the advantage of a folder is that it allows you to move a group of subsections at the same time, without changing their relationships with each other, in a single drag and drop.  The whole restructure, which felt as if I was turning the case for support inside out, took me a couple of hours.

Without Scrivener I couldn’t have written the case for support before the deadline. Two features were crucial: the ease of restructuring, and the rock-solid reliability with which I could create a perfectly formatted Word file after every edit.

  • Restructuring a case for support in Word is possible: you can use the outliner to move subsections around, but it is very hard to keep track of what is where, and I have never tried anything as complicated as this.
  • Preserving the format of lists and the styles of headings through an exercise like this may be possible in Word, but I have never managed to do it. I spend huge amounts of time renumbering numbered lists and at least once in an exercise like this I will have to recreate all the heading styles from scratch.

Scrivener has a couple of other features that make it useful, but not uniquely so.

  • It allows you to assemble all the background information you need and keep it together with the text in a file called the project. You can do this by using the folder structure of the file system.
  • It allows you to tag files in different ways so that it can produce several different output files from the same project. I used this to produce other documents, like the pathways to impact, the summary and the objectives.

Finally, there is one thing I would like Scrivener to do that I haven’t yet managed to do. I would like it to insert boilerplate text several times in a document. Followers of this blog will know that I like to repeat myself and that I will re-use a key sentence several times. If I could do this by referring repeatedly to a single sentence, so that all the instances would change if I edited any of them, that would be perfect. Then I would be able to write a whole case for support by numbers!

Life After Academia

More enjoyable work and less of it!

More enjoyable work and less of it!

It is almost exactly 2 years since Parker Derrington Ltd opened for business and over a year since I  changed career definitively and gave up my university job to become a full-time businessman.

The change of career was a leap in the dark but 3 key facts convince me that I have landed on my feet:-

  • I enjoy my work more.
  • I do less work.
  • I earn more money.

Of course, things could be even better and I want to use this blog to improve them. This post is a review of my career change and an outline of what I want to do better. It follows one of the good practices I developed as an academic manager – annual planning and target setting.

When I changed career,  the target I set myself was to develop enough paid work to replace my salary before the end of 2015.  I had a 3-point strategy:-

  • Start a blog;
  • Build a website;
  • Offer free workshops to people I knew and generous discounts to people that I didn’t.

My plan was that the blog would bring people to the website; the website would bring clients; and free workshops would turn friends into clients. I thought that any client that had a free or half-price workshop would very quickly order a full-price workshop once they knew how good they are.

I was wrong both about free work and about discounts. Clients act as if workshops are only worth what they pay. They act as if my workshops are worth nothing. The workshops are very good but the attendees seem to expect them to be bad. And, they don’t lead to more work: clients act as if they  are reluctant to pay full price for something they have had free or half-price. So now I charge the full price and I offer new clients a BOGOF (buy one get one free – two days for the price of one). For many clients the BOGOF clinches the deal. I am careful to make sure that my invoice states the full price and applies the discount so that the client sees the full price even if they don’t pay it. Since I changed my approach, a good proportion of clients have asked for follow-on work at full price.

The website and blog, which I promote through Twitter, have also been very useful. Of the 30 different clients or organisations that have hired me, about 20 found me through the web site or through Twitter.

I met the target I set myself with a healthy safety margin. I earned more last year through the company than I would have done if I had stayed in my job. However, although my income is higher than it was when I was an academic, it is a lot less predictable. I never know when I will get offered work and the delays between working and getting paid are variable and huge. The fastest payer so far was the University of Exeter, which paid within 3 days of my invoice. I won’t name the slowest payer because, even though they took over 4 months to pay, they will probably be replaced by someone even slower quite soon. Almost all my business is with universities and  almost all universities are apallingly slow to pay their bills.

My next target is to change the  mix of work. At present it is about 60% teaching people how to write, 25% writing and editing, 10% consultancy and 5% coaching.  I’d like to do more writing because that’s what I enjoy most. It’s probably also where I add most value. Very few clients trust me to co-write their grant applications and papers but those few are delighted with the results: better papers and grant-applications with less effort.

My target for 2016 is to  shift the balance of business towards writing and consultancy. My strategy will be to use the blog, the website and direct contacts to promote writing and consultancy.

So do get in touch if you don’t know what to write, if you don’t know how to write it, or if you know what to write and how, but don’t have time.

Seven Deadly Sins of Grant-Writing: Sins of Omission

3WiseMonkeysA couple of weeks ago I described deadly sins that grant-writers commit deliberately. This week I am dealing with sins that are just as deadly but much harder to avoid. The sins of omission just creep into your writing without you noticing and  you have to make special efforts to remove them.

The sins I want to deal with are Complex Sentences, Long Paragraphs, Poor Flow and failing to match the background to the project. They all meet  the definition of sin that I coined last week: “Anything that makes it hard for a committee member to pick up a clear understanding of the rationale of your research project, what it will discover and why that is important,  is a sin. So is anything that makes it hard for a referee to get a clear picture of the detailed reasoning in your argument and the detailed description of your intended research activities. Referees and committee both work under time pressure, so anything that slows them down is also a sin.”

Complex sentences are really difficult to avoid. They appear spontaneously in your draft. Most people can’t avoid writing them whenever they are trying to write something difficult – like a grant application.

That’s OK. Writing complex sentences isn’t the end of the world. Not unless your first draft is the end of your writing process. You must expect your first draft to be full of sins and you need to cast them out. You need to hunt through your draft and convert all the long, complex sentences into short, clear simple sentences. As a rule of thumb, you should redraft any sentence longer than 30 words or containing more than 1 verb or beginning with a digression – a phrase that is introduced by a word like “although”. And if it’s the first sentence of a paragraph you also need to make sure that the main message of the sentence fits on the first line.

It’s OK for complex sentences to appear in your first draft because that is usually the easiest way for you to write it.  But it’s not OK to leave them there. You have to replace them with simple sentences. This may involve breaking them up, or turning them round and it will take time, but you will get quicker with practice. Your final draft must be easy to read, and to speed read. Most of the people voting on your grant application will speed-read, or skim it. So if what you send them is full of complex sentences that have to be decoded carefully then they will not get your message, and you will have less chance of getting funded.

Long paragraphs are bad for two reasons.

  • I pointed out in my last post that most of the people scoring your grant will speed-read your case for support. Speed-readers read the first line of every paragraph provided there is white space between them. The longer your paragraphs, the less you communicate with speed readers.
  • Long paragraphs are usually very hard to digest. They are usually a sign that what you are writing is either very complex, or just a bit disorganised. The few readers who really want to read the detail in your case for support will find it hard.

If your paragraphs are longer than about 5 lines, try to break them up. If they are not too disorganised it will be fairly straightforward but if they are disorganised it may be easier to attend to the flow first.

Flow refers to the sequence of ideas that you present, sentence by sentence and paragraph by paragraph. Within paragraphs, good flow occurs when each sentence connects naturally to its successor. There are several ways of achieving this. If you have never thought hard about it (and I hadn’t until a few months ago), Google will find you countless sources of advice. I recommend that you read the Using English for Academic Purposes Blog, which has a section on paragraphs and flow.  The basic approach is that  you should always start the paragraph with the topic sentence, the one sentence that sums up the paragraph. Then, to  get good flow within the paragraph you make sure that the first sentence leads naturally to the start of the second sentence, which leads naturally to the subject of the third sentence and so on. This makes it easy for the reader to read through the paragraph without having to pause and analyse the wording to work out what you mean, or having to keep several ideas in mind in order to follow what you are saying.

Flow between paragraphs is also important and again Google throws up hundreds of ways to help you make it smoother. I think that the best approach here is to reverse outline, as suggested on the Explorations of Style blog, which is full of good advice on how to make your writing more readable.

Failing to match the background to the project is a sin against Derrington’s first commandment. You won’t go to hell for the sin but you may enter the purgatory of grant rejection. The commandment requires that before you describe your project and the outcomes it will produce, you use the background section to make the case that we need exactly those outcomes. It’s a pretty basic selling technique. It persuades the customer that they want what you are selling before you describe what you are selling. I have explained before how you use key sentences to create a structure that implements the technique by creating a background section that deals with the outcomes in the same order as the description of the project, and that explains, outcome by outcome, why we need them.

The key sentences also give you the best way to fix a mismatch between background and project. Basically you create the key sentences and then you use them to re-organise your text. And then you use them to write an introduction.

If you read a few successful grant applications you will realise that the sins are not fatal: most successful grant-writers commit them. However, the sins all make it less likely that you will get funded because they make it harder for time-pressed committee members and referees to do their job. Of course you may be lucky enough that the committee sees the merit in your application despite you making it difficult. But why take the chance?

Ten questions to Get Feedback on a Grant Application

FeedbackThis post was triggered by an  article in Harvard Business Review about how to get feedback on your performance at work. Getting feedback that you can trust is a universal problem and it affects grant-writers more than most. The key to getting good feedback  on a grant application is to think carefully about whom you should ask and how you should ask them.

Whom should you ask for feedback on your grant application?

The best option is to ask the right kind of expert, if you can. Expertise on grants is much more important than expertise in your subject. The ideal person would be someone who has served several years on the grants committee that will consider your proposal. Failing that, you could ask someone who has wide experience of similar grants committees and a good understanding of the scheme for which you are applying.  Unfortunately these people tend to be busy and you may be in a hurry. You can ask Parker Derrington Ltd to read your grant application and give you face-to-face feedback by Skype or Facetime, but you would have to pay. What can you do if you don’t have an expert on call and can’t afford to pay one?

If you can’t get an expert, you should choose an ignoramus, or someone who can act the part as far as your project is concerned. You need someone who does not share your views about what is important in your subject but who is open-minded. This mimics the kind of expertise that a grants committee member will have. The ideal person would be someone at your level, in a different but related field. It can be a good idea to exchange feedback with a colleague in a different subject area who is also preparing a grant application. It is a mistake to use someone who knows as much or more than you do about the area of your project, unless they are sufficiently experienced in the way grants committees work that they will be able to discount their knowledge of your subject.

How should you ask for feedback on your grant application?

If you ask a real expert for feedback, it doesn’t matter how you ask, they will give you useful feedback. Otherwise, you need to design your questions carefully if the answers are to be useful. Here are some suggestions.

  1. Write down the main outcome of the project. Mark where it describes it in the text with a (1)
  2. Mark with (2) in the text the section that gives evidence that the main outcome is important. Can you think of anything else that could be said to support the claim that the outcome is important?
  3. Is the importance of the outcome specific to this funder? Mark any references to the funder’s goals with (3).
  4. How many separate components does the research project have?
  5. Mark with (5-1), (5-2) etc, the sections of text that state the outcome of each component of the research project.
  6. Mark with (6-1), (6-2) etc, each section of text that explains how important the outcome is of a component of the research project.
  7. Mark with (7-1), (7-2) etc, the sections of text that give citations that support the explanations in (6-1), (6-2) and so on.
  8. Mark with (8-1), (8-2) etc, the sections of text that describe the research processes that will produce the outcomes (5-1), (5-2) etc.
  9. For each of these sections, can you mark with (?) any processes that are not clear, and can you suggest any extra processes that might be needed in order to produce the research outcome.
  10. Mark in the text with (1) where it states what will be done to make sure that the overall outcome of the project has maximum value to the funder. Can you think of any way that this could be made more valuable to the funder.

You should ask the reader to spend no more than half an hour answering these questions. If they are unable to answer them in that time then you need to do some serious rewriting. You might also consider reading chapter 11 of The Research Funding Toolkit, which has a comprehensive set of exercises to get feedback at every stage.

Finally, you should know that the correct answer to question 4 should be three.

We thank the referees for their helpful suggestions…….

PeerReviewCartoonIt may not be easy to keep your temper when responding to referees’ comments but you must make it seem like the easiest thing in the world. Whether the comments are on a grant application or a paper, you should compose your response with regard to the effect you want it to have on the reader. If you want your response to influence a grants committee to award your grant or a journal editor to publish your paper, anger is unlikely to help.

There are some differences between responding to grant-application referees and paper referees but most of the following recommendations apply to both.

Assume that the referee is trying to be helpful. You will do a better job if you can make the assumption of reasonableness. If you can convince yourself that the referees are doing their best to help you  and their comments are genuinely trying to help you improve your research project or your paper, it will be easier to follow most of the recommendations below. It doesn’t matter if the assumption is wrong and the referees are actually trying to sink your grant or turn your paper into tedious gibberish. The important point is that you will do a better job of responding to them if you assume that they are trying to help.

Take responsibility for the reader’s failure to understand. Some writers take pride in the knowledge that very few people understand their work. I have seen responses from grant applicants that berated referees for their stupidity. That is a risky attitude to take in any type of writing and is almost certain to lead to the failure of a grant application.

Express gratitude for referees’ suggestions. Derrington’s first law of responding to referees states “The less you feel gratitude, the more fulsomely you should express it”. This is probably more important in the case of a paper than a grant application because in the case of a paper the referee will usually be consulted about whether your response is satisfactory whereas in the case of a grant application your response will be usually assessed by the grants committee.  Of course it may help you to express gratitude if you can explain how the referees insightful suggestions have enabled you to transform the research project or paper.

Make it clear that the responding to the referees’ suggestions has enabled you to improve the research project (or paper). There are two reasons for this.

  • First, you want to give the impression that the final version of your paper (or grant application) is much better than the version that the referee evaluated and deserves a higher score than the referees gave it.
  • Second, you may want to distract the referee (or the grants committee) from the fact that you may not have made any changes at all. In this case, Derrington’s second law of responding to referees applies “The less you change, the more emphatically you state how much you have changed and how much this has transformed the paper.”

Derrington’s second law is more important for papers than for grants, because journal editors often regard referees’ recommendations as binding and authors often find them unacceptable. A grants’ committee is much more likely to recognise when a referee’s recommendation is ridiculous and accept a response that politely declines to implement it.

State clearly what you are responding to and how. This is particularly important when referees’ comments and suggestions are vague or ambiguous. It is often helpful to paraphrase the referees’ suggestions and to state what changes you have made in response to each one. This is a good way of dealing with the case when two or more referees suggest almost exactly the same thing. A list of your paraphrasings of the referees’ suggestions and a short statement about how you have responded to each one can make a very helpful executive summary that will reassure the reader that you have responded satisfactorily.

Some journals require you to do this but few funding agencies do. Even so, you should do it because it is likely to increase your chance of getting funded because it reassures the committee that you have responded satisfactorily to the referees’ criticisms.

Keep your overall response as short and simple as possible. This is more important with grant applications than with papers because the committee works under immense time pressure.

Know when to give up. If the referees’ reports on a grant are uniformly lukewarm it is unlikely to get funded whatever you say. Paradoxically, faint praise is more damning than strong condemnation because condemnation can give the impression that the referee is biased.

Of course, if you are overwhelmed by the desire to let the referees know how stupid their comments are, you probably won’t be able to follow any of the advice I have given so far. In that case I suggest that you wait until you can. Responding to referees is like inviting an  elderly relative to dinner, better not to do it than to do it with bad grace.

Say it again Sam. And use the same words.

groundhogRepetition of key sentences and key phrases is extremely important in a grant application. The key sentences that introduce each subsection of the background and the description of the project in the case for support should be repeated in the introduction and also in the summary.  So each key sentence should appear at least three times.

Some key phrases should be repeated more than three times because they occur in more than one key sentence. For example, imagine you are writing a grant in which one of the sub-projects will characterise the relationship between motherhood and apple pie.  The phrase ‘the relationship between motherhood and apple pie’ will be in two of your key sentences.  One will explain why we need to characterise ‘the relationship between motherhood and apple pie’.  The other will introduce  the description of the sub-project that characterises the relationship between motherhood and apple pie.

Most academics accept that it is helpful to repeat key sentences. But most of them reject the idea that the repetition should use  the same words in the same order.  So I want to explain now why it is more effective to use the same words in the same order whenever you repeat a phrase or sentence.

Effectiveness is much more important here than correctness.  Few would disagree with the assertion that exact repetition is a more correct use of English than paraphrasing but it is much more important to think about how you can increase the effectiveness of a grant application by using repetition in the way that I recommend and how you will fail to increase effectiveness in the same way if you change the words you use or their order.

In thinking about the effectiveness of a grant application, we should consider who will read it and how.  Committee members and referees have different needs and derive different benefits  from repetition.

The most important readers are the committee members that make the decision.  All of them will have a vote in deciding whether or not the grant application gets funded. Few, if any,  will understand the details of the research topic. All of them will read the summary and most of them will stop there. Some will try to read the application and understand it. Usually two  members of the committee, the designated members, are tasked with reading the application and leading the committee discussion. They will try hard to understand the application, but they will find it very difficult and they won’t have much time – maybe an hour. Any help you give them will be gratefully received.  Although most of the rest of the committee will not read the application they will probably glance through it during the discussion.

There are three ways that repetition is particularly helpful to committee members:-

  • Repeating the key sentences means that all the committee members will be likely to remember them. Even those who just glance through the application once will read the key sentences three times. This means that there is a very good chance that they will remember them and understand the logic of your case for support – what outcome your project will achieve, why it is important, what things you need to know in order to achieve the outcome and how you will achieve them. If you repeat the key sentences but substantially change the wording then people will be less likely to remember them. Every change in wording is likely to be interpreted as a change in meaning, leading to potential confusion.
  • Repeating key phrases in the sentences that state what we need to know and what the sub-projects will discover makes it very clear that the project will discover exactly what we need to know. In this way the key phrases act like labels for the different parts of the project.
  • Repeating the key phrases enables committee members to learn them and to have a sense of what they mean. Humans learn the meaning of new phrases by encountering them repeated in different contexts. Committee members who read your grant application carefully will get the sense that they know what it means , even if they don’t. If you vary the wording of the key phrases it becomes harder to learn them and  less clear to the reader that you mean the same thing.

The referees are, notionally at least,  experts in the research topic. They will read the application, write an analysis of its strengths and weaknesses and give it a score, which the committee will consider, but not necessarily follow.  The referees are likely to read your application more carefully than the committee members and to have a deeper understanding of the topic. However, they will want to assess whether the detailed content of your case for support actually supports the assertions made in the key sentences. Repetition helps them do this in two stages.

  • Reading the key sentences in the summary and in the introduction allows them to create a mental list of questions to which they want to find answers in the case for support.
  • Repeating the key sentences at the head of each subsection of the case for support guides the referees to the answers. Again, the key phrases act like labels. For example, a referee that has some doubts about whether your research approach really will characterise the relationship between motherhood and apple pie will be guided straight to the place where you describe the relevant part of your research approach by the phrase the relationship between motherhood and apple pie. If you decide to change any of the words in the key phrase, not only does it become less effective as a label, it also introduces the possibility that you are seeking to do a piece of research without having told the reader why it is important to do it.

In sum, repetition of key sentences and key phrases makes a grant application more effective in four different ways, provided that you use the same words in the same order.

 

 

Committees and Referees

Committees Like a Simple Story: thanks to Science and Ink http://www.lab-initio.com

Committees prefer a clear story: http://labinitio.com

The Journal Nature reported yesterday that scientists have complained that there is a mismatch between expert referees’ evaluation scores of research grant applications and funding decisions.  Different interviewees claimed that this mismatch either does or does not indicate either a flaw in the system or mistakes by referees or by committees.

There may be flaws in the system and mistakes probably happen, but there is a more obvious reason that referees’ scores should not be expected to predict funding decisions. Referees and committees do different jobs which impose different constraints on the way a grant application is written. Very few proposals are written in a way that satisfies both sets of constraints, and so, for the majority of proposals, there is no reason to expect a close match between the referees’ score and the committee’s score. Before I explain the constraints and how to meet them, I’ll clarify the story and explain its relevance.

The story was based on 302 grant applications to the Medical Research Council. It states that some applications that received high scores from the referees were triaged (rejected without being discussed by the committee). Of the proposals discussed by the committee, the story states that the group of applications that were successful and the group that were rejected had ‘a nearly identical spread of scores’. It’s worth noting that the story focuses on a statistic of the scores that is not particularly informative (spread) and does not mention any other statistics. It is relatively common that the referees include both friends and foes of the applicant, which can cause the spread of scores on a single application to range from the lowest to the highest possible. Consequently, nothing that the story says about this rather small data sample indicates a general failure of referees’ scores to predict committee decisions.

However, whatever we may try to conclude from this small dataset, most funding agencies (the EPSRC is an exception in the UK) ask referees and committees to do very different jobs. These jobs depend on different aspects of the way the proposal is written.

Referees work alone and each one works on a single application. The referees are expected to be experts in the same research field as the applicant and their evaluations are typically seen as coming from within that field. Their main task is to test the detailed rationale of the proposal. Are the research questions important? Is the research approach feasible? Does the research team have the ability to carry out the project? What is their standing in the field? Has their previous work made a significant contribution? Is the approach novel? Are other teams likely to get the answers first?

To get a good score from a referee, a grant application must contain relevant detail. Evidence that the questions are important and relevant must be cited. Any evidence to the contrary must be dealt with effectively. The research approach must be described clearly and in sufficient detail to convince a knowledgeable sceptic that the project is feasible, can be carried out with the resources requested, and will lead to the promised outcomes.

The committee works as a collective and takes a view across all the applications before them, typically about 100 for a single meeting. They must also take a view across all the different research fields that the committee supports. They need it to be clear that the project will deliver an  outcome that will have importance beyond the immediate research area and that the applicant has identified an approach that is likely to be productive and that the research team has the skills to deliver it. They need to know what the research aims are and how they relate to the overall outcome. They need it to be clear that the research objectives will satisfy the research aims and deliver the overall outcome and that the results will have appropriate impact

More importantly, the committee also works very quickly. They have to reach an agreed view about grant applications which most or all of them may not understand completely. For the committee to score an application highly they need it to be possible to understand it on the basis of a single hasty reading – or even a quick skim. The case for support should have a very clear structure that states clearly what will be the research outcome, why it is important, what are the research aims, what are the objectives and what will be done with the results.

Of course the best possible case for support has a clear structure that enables the committee to understand it and appreciate its strong points. It also uses the structure to guide the referees to the relevant detail. In my experience cases of support written in this way get very high scores both from referees and from committees. Unfortunately they are very rare but this blog explains in several different ways how to write them.