Overview
The methods of assessment for Quality of Science included the following activities:
• Listening to face to face scientific presentations (during visits to Centers and Value Chain Research Hubs)
• Discussion of presentations with scientists (during visits to Centers and Value Chain Research Hubs)
• Discussion of L & F Programme work with scientific collaborators, policy makers, delivery partners and beneficiaries (during visits to Centers and Value Chain Research Hubs) and additional interviews by telephone or Skype
• Interviews with managers of science, finance, quality assurance and scientific staff (during visits to Centers and Value Chain Research Hubs) and additional interviews as needed
• Review of the of bibliometric assessment provided by IEA
• Reading and scoring a random sample of 25-50 percent of peer reviewed publications including the “top five” best outputs provided by L&F Programme leaders.
• Reading and scoring a random sample of 20 percent of non-peer reviewed publications (see table below for a definition of “publication”)
• Assessment of case studies produced by the review team during the assessment
Bibliometric analysis
The analyses below were conducted based on a list of publications provided by L&F. The list included publications produced pre-CRP (2010-2011) and post CRP (2012-mid2015), whereas the numbers of publications in the database were much lower for the pre CRP period.
Citations were sought using Google Scholar for all Journal articles published in the period from 2010 to 2015. Impact factors of journals in which L&F published are based on Journal Citation Reports (JCR) from 2014. The cut-off date for this was 31 August 2015.
H indexes of the leadership team were sought using both Google Scholar and Scopus, whereas the information presented in the main body of the report refers to the Scopus data. The cut-off date for the H indexes is 1 Oct 2015.
Scoring of outputs
A total of 223 (79 peer reviewed and 144 non-peer reviewed) publications were assessed.
Documents for scoring were selected from a database of 829 published outputs. This was a subset of the database of 2019 provided by L&F after screening to remove items not considered to be
scientific outputs (brochures, internal reports and some posters, presentations and wiki items that
21
Evaluation of the CRP on Livestock and Fish - ANNEXES
cgiar.iea.org were announcements of intent rather than actual outputs). A few items were also excluded because
they were published in a language that the evaluators could not read, or could not be accessed.
The list was stratified by discipline area and then by peer reviewed/non-peer reviewed as defined above. For the purposes of the evaluation, outputs were considered to be peer reviewed if they were published in journals, books or theses where they had clearly been subjected to an external peer review. All others were considered to be non-peer reviewed. The team acknowledges that published reports and briefs are subjected at least to internal peer review, but there was insufficient information in the database provided to assess the level of reviewing that had been applied. Five key outputs in each discipline area, indicated by FP leaders, were included for scoring and the remainder of the sample was selected at random from within each sub-stratum. This means that there may be a slightly favourable bias in the results, since research leaders could be assumed to present work they considered to be their best, but it also ensured that the evaluators did not inadvertently miss reviewing important outputs.
Scoring was done using a 3 x 3 matrix. This is demonstrated in the diagram below. Scoring was undertaken by members of the review team with expertise in the relevant scientific “discipline areas”. At least 25% of the peer reviewed outputs were scored for all “discipline areas” and in many areas at least 50% were assessed.
Each publication, whether peer reviewed, or non-peer reviewed (grey press/report/powerpoint presentation), was be scored on two criteria, a) overall Quality and/or Novelty and b) Impact and/or Usefulness. For consistency, the same scoring system was used for peer-reviewed and non peer- reviewed outputs.
a) overall Quality and/or Novelty is scored from C-A with C being acceptable, B being good and A being excellent.
b) overall Impact and/or Usefulness is scored from 1-3 with 1 being acceptable, 2 being good and 3 being excellent
If any publication failed to meet the minimum acceptable scored for either Quality/Novelty or Impact/Usefulness then the publication was recorded as not meeting the standard.
For peer reviewed publications, Quality was defined as meeting international or national standards of rigour for study design, methodology, interpretation of results, presentation of hypotheses and conclusions from the research. Impact was broadly defined as the benefit, or potential benefits, from the research described for the discipline or research area.
For non-peer reviewed publications (grey press/report/powerpoint presentations) Novelty was described as the originality of the publication in it’s aims and objectives and the appropriateness of the study design. Usefulness was assessed by the potential for uptake and use of the output in the context of the target reader or user of the information.
Nine scores were possible for each peer reviewed and non-peer reviewed publication/outputs: 1C, 2C, 3C, 1B, 2B, 3B, 1A, 2A, 3A.
22
Evaluation of the CRP on Livestock and Fish - ANNEXES
cgiar.iea.org Quality/novelty
Acceptable (1) Good (2) Excellent (3)
1A 2A 3A Excellent
(A)
Impact/usefulness
1B 2B 3B Good
(B)
1C 2C 3C Acceptable
(C)
In addition, each publication whether peer reviewed or non-peer reviewed, was scored (yes/no) for the following criteria which are considered important to L & F:
• Inclusion of gender issues
• Relevance to poverty alleviation
• Relevance to food security
• Evidence of multi- or inter-disciplinary research
23 cgiar.iea.org
Evaluation of the CRP on Livestock and Fish - ANNEXES