DORA Releases New Guidance on Assessment of Research/Researchers

The Declaration on Research Assessment (DORA) has issued new guidelines to address the misuse of quantitative indicators in research evaluation. Building on its initial principles to move beyond the Journal Impact Factor, DORA’s new document provides advice on using metrics like the h-index, citation counts, and altmetrics.

Emphasizing clarity, transparency, specificity, context, and fairness, the guidance aims to prevent oversimplification in research assessments:

Be Clear: What is your rationale for using particular quantitative indicators in your research or researcher
assessments? Is it grounded in good evidence?

Be Transparent: Ideally, rules for the use of quantitative indicators in research assessment should be developed in dialogue with your research community. They should be published so that those being evaluated understand your criteria. Make sure also that reviewers are fully aware of your approach to using quantitative information in
assessment.

Be Specific: How well does the indicator refer to the qualities of the person or the piece of work being assessed? Be mindful of aggregate metrics (e.g., JIF, h-index), which conceal large variations in performance, and of composite indicators (e.g., scores in university league tables, altmetrics), which are made up of arbitrarily weighted scores for very different attributes and activities and are therefore difficult to interpret meaningfully.

Be contextual: How will you take account of the proxy and reductive nature inherent in any indicator? (E.g., citations are not a direct measure of quality; the h-index takes no account of age, discipline, or career breaks.)

Be Fair: How will you avoid biases inherent in quantitative indicators? Though it is often assumed that bibliometric indicators are “objective,” decisions to publish a paper or to cite it are choices that can reflect structural and personal biases. Decision makers need to be proactive and transparent in efforts to mitigate the impact of these biases in research assessment — and the same obviously applies to the qualitative aspects of assessment.

They explain how these principles apply to some of the more commonly used indicators such as the Journal Impact Factor (JIF), citation, h-index, altmetrics. The list is not exhaustive, but it is hoped that these examples will show how the principles can be applied in practice to any quantitative indicator.