Scientific Peer Review Process for Research Grant Competitions,
Institute of Education Sciences

Peer review at the Institute of Education Sciences (IES) begins with the requirements articulated in the Request for Applications (RFA). Each specific competition has its own RFA. The RFA includes the factors that will be considered by the scientific peer review panel.

General criteria and initial screening

The general criteria that currently apply to all research grant competitions are the significance of the proposed project, quality of the research plan, qualifications of the personnel, and degree of access to resources required to conduct the research. In addition, each RFA describes the content focus of the competition and details the methodological expectations for the proposed research. IES identifies potential reviewers who have the scientific expertise to evaluate the proposals on these critical factors, including content expertise (e.g., reading comprehension or teacher professional development), research methodology, and data analysis. Potential reviewers are identified primarily on the basis of quality of the research they have conducted and published in scientific peer-reviewed journals.

Similar to the National Institutes of Health, IES has separated responsibilities and staffing for the offices that oversee research programs from responsibilities and staffing for peer review. Thus, staff members who are responsible for overseeing research programs are free to interact with applicants without potentially biasing the selection of the peer review panels in ways that might favor one applicant over another.

Applications are screened initially for compliance with the requirements of the RFA. Screening occurs for matters of substance, such as whether the methodological and content requirements stipulated in the RFA have been met. Screening also occurs for compliance with procedural rules such as page length, formatting, and the content of appendices and attachments. Applications that are identified as being out of compliance in the initial screening are not passed on to peer reviewers for consideration. A second, more intensive examination for compliance occurs after peer review and prior to a funding decision.

Each application that has passed the screen for compliance is assigned to at least two primary reviewers who complete written evaluations of the application, identifying strengths and weaknesses related to each of the four general review criteria: significance, research plan, personnel, and resources. For each application they review, primary reviewers independently assign a score for each criterion, as well as an overall score. Based on the overall scores assigned by primary reviewers, an average overall score for each application is calculated and a preliminary rank order of applications is prepared before the full peer-review panel convenes to complete the review of applications.

Peer-panel review

The full peer-review panel consists of the primary reviewers for all of the applications under a particular RFA. A typical full panel is 15 to 20 members, in order to facilitate discussion of the applications. Sometimes the panel is augmented by ad hoc reviewers when more than 20 reviewers are needed for primary reviews. Ad hoc reviewers are not considered members of the panel and participate in the panel discussion only for applications for which they were primary reviewers.

Panel members may not submit proposals to the competition for which they are reviewers. In addition, reviewers may not serve as a primary reviewer for and must recuse themselves from participating in the panel discussion and scoring of any application from individuals with whom they have a close relationship (e.g., collaborators on current projects or former students) or from the institution at which they are employed.

Panelists meet together for two days to consider the top-ranked applications from among the total pool of submitted applications. One of those panelists, selected ahead of time, serves as chair. Staff responsible for overseeing the research program under which a competition is being held attend the panel meeting as observers.

In order to allow panelists time for careful attention to those proposals that have a chance of being funded, the full panel considers only those applications deemed to have the highest merit. That is determined by the preliminary rank order derived from each application’s average overall score from the primary reviewers. Generally, the top-ranked 25 to 30 applications are sent forward to the full panel for review. In addition, a panel member may nominate for consideration by the full panel any proposal that he or she believes merits full panel review but would not have been included in the full panel meeting based on its preliminary rank order. The end result is that the most competitive proposals are reviewed by the full panel.

At the panel meeting, each of the top-ranked applications is presented to the panel by the primary reviewers. The lead primary reviewer describes the application and identifies its strengths and weaknesses. The second primary reviewer provides any additional comments. The panel members then discuss the merits of the application based on the primary reviewers’ critiques and their own appraisal of the application. After the discussion, each panel member independently rates the application. After the panel meeting, the scores for each application are averaged across reviewers, and a final rank order is determined.

The applications considered by the full panel receive overall quality scores on a scale that ranges from 1.0 to 5.0 (1.0 to 1.5 = outstanding; 1.6 to 2.0 = excellent; 2.1 to 2.5 = very good; 2.6 to 3.5 = good; 3.6 to 5.0 = unacceptable). Generally, IES only considers for funding those applications that the panel has rated as being outstanding or excellent (on this scale, 2.0 and below).

Funding decisions

As indicated in the RFAs, the award decision is based on scientific merit, responsiveness to the requirements of the RFA, performance and use of funds under a previous federal award, contribution to the overall program of research described in this request, and the availability of funds. In practice, IES generally funds down the slate to the cut point of 2.0 on the final panel ratings of the applications after a thorough review to determine that each application has met all the requirements of the RFA. Maintaining high research standards is important in the context of IES's overall mission, a key aspect of which is to improve the quality of education research in the United States.

Since the implementation of the new peer review process at IES, the number of competitions has grown from 3 in Fiscal Year (FY) 2002 to 11 in FY2005. The number of applications received has increased from 226 in FY2002 to 661 in FY2004. Finally, the number of reviewers has grown from 73 to 162. Currently, IES competitions on particular topics (e.g., reading comprehension) occur once each year. In the future, IES may have semiannual competitions for those recurring competitions that generate a large number of applications.

For more information about the Association's resources on No Child Left Behind and related policy issues, please contact the Public Information Office of the International Reading Association, 800 Barksdale Rd., PO Box 8139, Newark, DE 19714-8139, USA. Phone: 302-731-1600, ext. 293; fax: 302-731-1057; e-mail: pubinfo@reading.org.