nih rfi & my eight opinions.

NIH put out a request for information asking, essentially, how it should figure out what good science looks like.

Officially it’s NOT-OD-26-087, “Measuring and Rewarding Scientific Impact.” Unofficially it’s NIH asking the entire biomedical research community to weigh in on the thing that quietly runs everyone’s career: what counts.

I’ve spent enough of this project’s marginalia complaining about how publication has become a stand-in for scientific value that it felt dishonest not to answer when someone with the power to actually change it asked directly.

So I wrote back. At length. In eight parts, because that’s how the RFI was structured and because I apparently had opinions about all eight.

eight things, roughly

Rigor and reproducibility — replication, negative results, and reanalysis should count as science, not as the absence of it.

Data, software, and models — the tools thousands of labs depend on are usually built and maintained by a handful of people nobody’s tenure committee has heard of.

Training and mentorship — trainees don’t all become PIs anymore, and mentorship should be judged by whether it prepared them for the job they actually got.

Collaboration — team science still gets evaluated as if only the last author on the paper did anything.

Translation — bench-to-bedside is real, but so is bedside-to-bench, and NIH’s incentives mostly only see the first direction.

Foundational exploration — some of the most important work looks unproductive for years before it looks essential.

Public impact — outreach shouldn’t be measured in headcounts at a science fair; it should be measured in relationships that last.

Publishing itself — preprints, open review, and Publish-Review-Curate models deserve to count as scholarly output, not just as a preview of the “real” paper.

my response, in full

Response to NIH Request for Information on Measuring and Rewarding Scientific Impact (NOT-OD-26-087)

Rigor and Reproducibility

Scientific rigor should be recognized as a primary research output rather than a compliance requirement. While biomedical research has traditionally rewarded novelty, equally important activities, including replication studies, validation experiments, publication of negative or null findings, benchmarking analyses, and methodological refinement, remain undervalued despite their essential role in building a trustworthy scientific literature.

NIH should develop measurable indicators that recognize contributions to reproducibility, including publication of independent replication studies, release of reproducible computational workflows, transparent reporting of methods and quality-control metrics, reanalysis of publicly available datasets, and development of benchmark datasets or community standards. Well-designed studies that produce negative or null results should be recognized as valuable scientific contributions because they prevent unnecessary duplication, refine biological hypotheses, and improve future experimental design.

To further incentivize rigorous science, NIH should consider dedicated funding mechanisms for replication studies, administrative supplements to validate influential findings, and grant review criteria that explicitly reward reproducibility efforts. Expectations should remain flexible across disciplines and career stages while recognizing that rigorous science is an ongoing process of refining knowledge rather than simply producing novel discoveries.

Data, Software, and Model Sharing

Data, software, computational workflows, and predictive models have become foundational products of modern biomedical research. Yet these resources often receive substantially less recognition than traditional publications despite enabling thousands of downstream discoveries.

NIH should explicitly recognize open-source software, databases, computational workflows, interoperable standards, and publicly accessible datasets as primary scholarly outputs. Indicators of impact should include software adoption, community contributions, dataset reuse, workflow implementation, API usage, long-term maintenance, and compliance with FAIR (Findable, Accessible, Interoperable, Reusable) data principles rather than publication citations alone.

Importantly, NIH should establish dedicated funding mechanisms to support the development, maintenance, and long-term sustainability of open-source scientific software and infrastructure. Many of the most widely used computational tools in biomedical research are developed and maintained by small research groups with limited long-term support despite serving thousands of investigators worldwide. Sustaining these resources should be recognized as a valuable scientific contribution, not merely technical service.

Training and Mentorship

The future strength of the biomedical research enterprise depends not only on producing excellent science but also on preparing scientists for the increasingly diverse roles they play in society. Modern scientists serve as mentors, educators, communicators, entrepreneurs, software developers, policy advisors, and community partners in addition to conducting research.

NIH should broaden how mentorship and training are evaluated by recognizing contributions that prepare trainees for this evolving landscape. Potential indicators include trainee career outcomes across academia, industry, government, education, policy, entrepreneurship, nonprofit organizations, and clinical practice; evidence-based mentoring practices; trainee independence; structured mentorship programs; and mentoring climate.

Training programs should also incentivize skills that are increasingly essential to modern science but are often treated as optional. These include science communication, science policy, community engagement, entrepreneurship, open science practices, data stewardship, peer review, and interdisciplinary collaboration. Investigators who train students in transparent scientific practices, including preprinting, reproducible computational workflows, responsible data sharing, accessible communication, and constructive peer review, should receive formal recognition for these contributions.

Scientific excellence should include preparing the next generation of scientists not only to conduct outstanding research but also to effectively communicate, translate, and steward that research for the benefit of society.

Collaboration

Biomedical research increasingly relies on interdisciplinary teams that span institutions, sectors, and scientific disciplines. However, many evaluation systems continue to emphasize individual accomplishments rather than collaborative contributions.

NIH should recognize meaningful contributions to team science regardless of publication authorship position by considering leadership of collaborative projects, development of shared infrastructure, multi-investigator grants, co-mentorship, community resource development, and documented contributions through narrative team science statements or contributor role taxonomies.

In addition, NIH should expand funding mechanisms that intentionally build scientific capacity through collaborations with Minority Serving Institutions (MSIs), Historically Black Colleges and Universities (HBCUs), Tribal Colleges and Universities (TCUs), community colleges, primarily undergraduate institutions, and emerging research institutions. These partnerships should be structured as reciprocal scientific collaborations that build long-term research capacity rather than short-term subcontracting arrangements.

Such investments would not only strengthen scientific discovery but also broaden participation in biomedical research while expanding research capacity throughout the United States.

Entrepreneurship and Translation

Translation should be viewed as a bidirectional process rather than a one-way pathway from bench to bedside. Basic scientists should have greater opportunities to move discoveries toward clinical implementation, while clinicians, patients, engineers, public health practitioners, policymakers, and industry partners should have meaningful opportunities to inform fundamental biological discovery.

NIH should recognize diverse forms of translation, including clinical implementation, public health impact, policy translation, development of diagnostics and research tools, open-source technologies, nonprofit partnerships, educational innovations, and commercialization where appropriate.

Expanding funding mechanisms that intentionally connect basic science laboratories with clinicians, public health practitioners, patient organizations, entrepreneurs, and industry partners would strengthen the entire translational ecosystem. These partnerships should be viewed as complementary approaches for maximizing the public benefit of biomedical research rather than competing models of scientific impact.

Foundational Scientific Exploration

Many of the most transformative advances in biomedical research emerge from high-risk, high-reward investigations that require sustained investment and may not produce immediate publications or translational outcomes. Current incentive structures often favor predictable projects with short-term deliverables over ambitious questions capable of fundamentally changing scientific understanding.

NIH should continue expanding support for foundational scientific exploration by recognizing long-term infrastructure development, technology innovation, biological resource generation, atlas-building efforts, methodological advances, and exploratory research that creates new scientific directions.

Review criteria should emphasize scientific significance, originality, and long-term potential rather than short-term publication metrics alone. Investigators should not be penalized for pursuing ambitious projects whose greatest impacts may emerge years after the initial funding period.

Public Impact

The ultimate purpose of biomedical research is to improve human health and benefit society. Accordingly, public impact should become a core dimension of scientific excellence.

Public impact should extend beyond counting outreach activities or media appearances. Instead, NIH should recognize sustained, reciprocal relationships between scientists and the communities they serve.

Potential indicators include long-term partnerships with patient advocacy organizations, community advisory boards, collaborations with K-12 schools and community colleges, publicly accessible educational resources, contributions to evidence-informed policymaking, citizen science initiatives, implementation of research findings into clinical practice or public health, and measurable efforts to strengthen public trust in science.

NIH should also recognize contributions that improve the accessibility and transparency of science itself, including open educational resources, accessible software, publicly available datasets, preprints, transparent communication of findings, and efforts to make biomedical knowledge broadly understandable and reusable.

The goal should not simply be increasing public awareness of science but strengthening enduring relationships between scientific institutions and the communities they exist to serve.

Other Areas

One area deserving additional consideration is modernization of scientific publishing itself. The current publication system remains one of the strongest drivers of scientific incentives, yet many activities essential to scientific quality remain largely invisible during evaluation.

NIH has an opportunity to encourage a transition toward more open, transparent, and efficient models of scientific communication by explicitly recognizing preprints, open peer review, and emerging Publish, Review, Curate (PRC) models. These approaches separate dissemination, evaluation, and curation into distinct scholarly activities, allowing discoveries to become immediately accessible while expert review and long-term curation continue transparently.

NIH should consider allowing investigators to document preprints, citable peer reviews, open review activities, and community curation efforts as evidence of scholarly impact. Likewise, reviewer contributions, editorial leadership, and development of community standards should be recognized as meaningful scientific outputs rather than invisible service activities.

Finally, NIH should consider adopting a broader Scientific Impact Portfolio that complements traditional publication metrics. Such a portfolio would allow investigators to document contributions across scientific discovery, reproducibility, software and data infrastructure, mentorship, collaboration, translation, public engagement, scientific citizenship, and open science. This approach would better reflect the collaborative, interdisciplinary, and mission-driven nature of modern biomedical research while creating incentives that strengthen the scientific enterprise as a whole rather than rewarding only individual productivity.

Sincerely,

JP Flores, PhD

why this one felt different to write

The 2 CFR Part 200 comment was defensive. It was me trying to stop something from happening.

This one was the opposite. NIH didn’t have to ask. It asked anyway.

That’s not nothing. An RFI is, structurally, an agency admitting it doesn’t already have the answer.

I don’t know how much of this makes it into whatever comes next. But the request for information was real, and so was my answer to it.