While good, ICMR impact scale may deter research in public interest

The Indian Council of Medical Research (ICMR) recently proposed the Impact of Research and Innovation Scale (IRIS), a scale with which to measure the impactfulness of the effects of biomedical, public health, and allied research projects funded by the organisation. ICMR is India’s topmost governmental grants-giving and research agenda-setting body vis-à-vis health research, so the scale and its calculi will be of great consequence to the Indian medical and health research communities.

ICMR proposes to measure research impact in units called publication-equivalents (PEs). A research paper published in a peer-reviewed journal that reports results or methods of primary research, or a systematic review and meta-analysis is assigned 1 PE — while a research paper that is cited in policies/guidelines is assigned 10 PEs. A patent’s impact is 5 PEs and that of a commercial device being used at scale is 20 PEs.

Pros of standardisation

There are several advantages to measuring research impact in this way. First, using PEs as the ‘unit of impact’ provides a standardised frame of reference to discuss different kinds of impact. It will allow ICMR to assess the impact of heterogenous work from researchers across disciplines, such as biochemistry, physiology, biomedical engineering, public health, etc., operating at different scales, including basic science, translational sciences, population public health sciences, etc.

Second, IRIS recognises that there is more to research impact than citations. Medical and health researchers, particularly those working in academic setups, typically lack incentives to pursue research that wouldn’t lead to citable academic papers. But a scale like IRIS could break that structure and incentivise researchers to diversify scholars’ research portfolio. Third, tying PEs and IRIS to actual decisions about funding allocation and project prioritisation ensures it won’t be a theoretical,  academic exercise. That ICMR is piloting IRIS to measure the impact of existing research grant programmes and institutes is an important signal of action.

Skewing research evaluation

For the same reasons, it’s important to examine where and how this relatively simple approach to standardising research evaluation could fail. First, a sound theoretical rationale is missing for PEs as a unit of research impact the way the ICMR will wield it. The ICMR note states that commentary, perspective, and narratives review papers will have 0 PE. In this case, the 1977 paper that introduced the biopsychosocial model of medicine, which transformed medical and public health research, will have no impact. Articulating new ideas and critical discourse around emerging evidence are at the foundation of research and are evidently impactful, yet the PE-based system could discourage Indian researchers from pursuing articles of this nature.

Second, IRIS can skew how certain research is valued over some other. Research that leads to policy changes receives 10 PEs while commercial devices receive 20 PEs. As a result, the RATIONS clinical trial that studied nutrition in tuberculosis patients and India’s Home-Based Neonatal Care that revolutionised community health programming will be deemed to be less impactful than, say, a commercialised robotic surgery device. This outlook could discourage basic science and academic medicine and coerce researchers and institutions to commercialise their research and innovations.

While it is their prerogative to do so, there is a danger of this happening at the expense of the ethos of scientific research as a public good. The merits of new proposals are contingent on the local context. Biomedical and health research in India are known to have suffered from a poor research ethics culture, and there is a risk this could expand to affect PEs as well. It is thus vital that our public sector institutions hold the line and stand with the ethos of ‘research as a public good’.

Finally, a unit and scale that will determine the fate of biomedical and health research in India should be developed following rigorous research standards, with full transparency and mechanisms for accountability. The ICMR note mentions an ongoing pilot across its institutions and positive feedback, but it should follow well-established study design and analysis methods to develop such scales. For instance, the assignment of PEs to different indicators could be done through a national-level Delphi study where researchers form a consensus on the assignment. Data must be shared with independent groups to analyse and validate the scale.

Measuring research impact is a complicated exercise with no correct answers. Institutions such as the ICMR have the opportunity and the responsibility to solve it creatively and collaboratively.

Siddhesh Zadey is a co-founder of the nonprofit Association for Socially Applicable Research (ASAR) and a doctoral student in epidemiology at Columbia University.