Evaluation of the NIH Website Usability

Executive Summary

• The NIH is the largest funder of health-related research.
• We evaluate the usability of the NIH website.

Introduction

The NIH website is primarily of interest to those that are applying for grants to perform health-related research. The NIH’s budget is enormous, as is explained in the following quote.

As NIH’s budget has grown to $30 billion annually—fully half of all civilian R&D expenditures – Issues.org

Think of that for a moment. One-half of all civilian R&D is spent by just one entity, the NIH.

As NIH grew, so did the institutions it funded to do research. NIH-funded research, is an industry that sustains academic health centers throughout the nation—and fuels ambitions among every research university to attract a bigger piece of the pie. Academic health centers have expanded remarkably over the decades, and entire careers are devoted to biomedical research lobbying. – Issues.org

Our References for This Article

If you want to see our references for this article and related Brightwork articles, visit this link.

About

This is the main introductory page for the NIH. This page is decently done and follows the standard tile approach to offering the options. 

This is the page that guides the user through the grant process. 

Here the website describes the budget for research of the NIH. 

This page explains the different institutes within the NIH. This page is not bad, but it is a bit long and could be shrunk and made easier to read. 

Back to how to apply for a grant. This information is quite lengthy, and one wonders why this is not simply included in a form, which gets into a primary problem with the NIH website. 

This page shows how much work is involved in submitting a grant proposal to the NIH. The advice is quite lengthy but could be included in a form. We have done this ourselves by running a survey using Google Forms and including all of this information in the form. 

As with the FDA, the NIH very much relies upon PDFs. However, in this case, rather than publishing information in PDFs, the NIH uses PDF forms.

I am very unfamiliar with PDF-based forms, and I think this applies to others as well. Forms are so effectively run or housed within web pages, and, curiously, the NIH relies on PDF forms. This creates much more overhead and works for both the NIH and the grant submitters.

I don’t think many companies continue to use PDF forms. I don’t recall running into them. Having a form as a PDF makes sense if the person filling out the form will fill out a paper form. At one time, submissions for grants were made through the mail. One wonders if these forms started as paper and were migrated to PDF documents rather than being put into a web form.

Here is one of the PDF forms where you enter your basic information. 

The NIH forms also use attachments so that the attachment is then placed within the PDF and then sent to the NIH. 

I have no idea when or why the NIH decided to follow this approach.

• It does make sense to allow for attachments as it may allow for more complex and lengthy documents that can be entered into a form.
• It makes no sense to add attachments to a PDF.

Reviewing the NIH’s Forms and Their Problems

The forms the NIH offers are very unappealing and unnecessarily complicated and confusing to fill out. Forms within a web page are so much easier to set up and to fill out. Furthermore, these fields can go immediately into a database, allowing the data from the grant applications to be more effectively managed.

Another problem with the PDF forms is that there is little guidance within them, and they are cryptic. The NIH keeps the PDFs as small as possible, with the advice in the external documents (which I showed a screen show to earlier) and not within the PDFs.

This is very confusing and inefficient. Advice is easier to follow when it is right next to the field to be filled out.

If the grant application needs to make changes, a new form has to be sent (and there are many forms) rather than just updating the overall web form, which can be edited endlessly.

Constant Virtue Signaling at the NIH Website

A section of the NIH website talks about how they don’t tolerate sexual harassment.

That is good to hear, but why do the website readers need to know this?

The NIH can enforce its sexual harassment policies without the public needing to know. Secondly, isn’t the entire federal government required to comply with its anti-sexual harassment policy? That is, is sexual harassment tolerated in the FDA or DOT, but not the NIH.

Furthermore, This grandstanding takes the website away from its focus, making it easy to submit grant proposals. So while the NIH will not tolerate sexual harassment, they will tolerate exceptionally poorly thought out and inefficient grant submission technology.

Given their shortcomings with the grant submission technology, it seems as if they need more attention on making their website more usable for its main purpose. This is the primary way that those that submit grants to the NIH will interact with the agency. The NIH does not want people to send in printed-out copies of grants as that is even more overhead.

Conclusion

The simple parts of the NIH website, where they introduce themselves and provide high-level information, are ok, but where the website falls down is its primary purpose, which is to allow for the submission of grants.

In this area, the website relies on a large number of PDF forms which are not only inefficient and unnecessarily time-consuming for grant applicants to fill out but place a lot of administrative overhead on the NIH. It also means that the grant applicants must send their PDFs through email. It reduces the ability to edit the grant submission, track the grant submission, and then for the NIH to evaluate the proposal.

I can’t say how the NIH gets the data into a database, but collecting the data makes this process far more complex. This website shares some similarities with the FDA’s website. The focus appears to be more on communicating positive feelings and PR to the visitors to the website than serving its actual function.

We rate the NIH website a 2.5 out of 10 for usability, findability of and grand submission on the site. The primary reason is that there is just no excuse for the NIH to have such an inefficient mechanism for submitting grants.

After reviewing the NIH site, it was interesting to find the following article on the effectiveness of the NIH grant process.

The findings suggest that the expensive and time-consuming peer-review process is not necessarily funding the best science, and that awarding grants by lottery could actually result in equally good, if not better, results. A report on the research, published online Feb. 16 in the journal eLife, was written by Ferric Fang, MD, at the University of Washington, Anthony Bowen, MS, at the Albert Einstein College of Medicine and Arturo Casadevall, MD, PhD, at the Johns Hopkins Bloomberg School of Public Health,

“The NIH claims that they are funding the best grants by the best scientists. While these data would argue that the NIH is funding a lot of very good science, they are also leaving a lot of very good science on the table,” says Casadevall, Professor and Chair of the W. Harry Feinstone Department of Molecular Microbiology and Immunology at the Bloomberg School. “The government can’t afford to fund every good grant proposal, but the problems with the current system make it worse than awarding grants through a lottery.”

Notes Fang, a professor of laboratory medicine and microbiology at the University of Washington: “We are not criticizing the peer reviewers. We are simply showing that there are limits to the ability of peer review to predict future productivity based on grant applications. This suggests that some of the resources and effort spent on ranking applications might be better spent elsewhere. While the average productivity of grants with better scores was somewhat higher, the differences were extremely small, raising questions as to whether the effort is worthwhile.”

NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, NIH relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds. In recent years, the NIH has only funded those proposals ranked around the top 10 percent. The annual research budget for the NIH was $30.1 billion in 2015.

For their study, the researchers reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008. Researchers who published a paper in the journal Science in 2015 had collected the data set. Their research suggested that peer review did in fact work — that the highest ranked research projects funded by the NIH earned the most citations. – ScienceDirect

Therefore, the PDF forms that the associated explanatory information at the NIH website is just the very beginning of a highly dysfunctional process.

The researchers in this case chose to measure the success of a research grant by determining how many papers that resulted from the work were published in scientific journals and then tracked how many times those papers are cited in future research papers.

This appears to be Google’s approach to determining whether research should be funded. That is, they are relying on peer-reviewed journals. However, there is little evidence that peer-reviewed journals lead to better science, as I cover in this article on peer review.

The quote continues.

The original researchers looked at all of the grants funded by NIH in those years and a significantly larger number of grants were funded in many of those years. The percentage of grants funded in recent years has been at historic lows because of cutbacks resulting from sequestration budget cuts stemming from the October 2013 government shutdown.

For the new study, Casadevall and his colleagues decided to only look at the top 20 percent of grants awarded and found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to which would go on to be the most-cited research. What the peer review process can do, they determined, is discriminate between very good science and very bad science –that is, those in the top 20 percent versus those below the 50th percentile.

Peer review isn’t cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend a lot on peer review. That money could go toward more grants, the researchers say. The costs are not only financial. Writing and reviewing grants is extremely time consuming and diverts the efforts of scientists away from doing science itself.

The process also allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.

“When people’s opinions count a lot, we may be doing worse than choosing at random,” Casadevall says. “A negative word at the table can often swing the debate. And this is how we allocate research funding in this country.”

To solve this, the authors suggest that the top proposals are first chosen by peer review and that those proposals then be put into a lottery, with grants awarded at random. Lotteries were used as part of the military draft during the Vietnam War and are used today to fill magnet schools with many qualified applicants and to award permanent residency applications. College student and low-income housing is often awarded by lottery. He says New Zealand has started using a lottery to make its scientific grants.

Adds Casadevall: “We’re hoping people will look at this data and say, ‘Can we do better? Can we create a fairer system that gives society the best science it can afford?'”

More Interesting Quotes on the Research Orientation of the NIH

Despite its name, NIH’s mission has not generally been current health per se, but rather research for tomorrow’s health, and progress against intractable diseases through better understanding. An agency devoted to current health would do well to focus on tobacco control, exercise, nutrition, sanitation, and more cost-effective delivery of health care—prevention and efficiency, rather than research on diseases currently not treatable. NIH does these things—some programs such as the National High Blood Pressure Education Program and the National Cancer Institute’s ASSIST program have been signal successes in achieving health gains—but health and health care are not NIH’s main show. NIH is primarily about addressing diseases currently refractory to treatment, in hopes of changing that fact. Such questions are especially pertinent in light of the nation’s continually mediocre public health outcomes, and their stark contrast to the sophistication and productivity of the biomedical research enterprise.

Michael Crow, president of Arizona State University, wrote in Nature three years ago about how health research was unduly decoupled from health outcomes, and called for re-thinking how NIH and other components of biomedical research might more directly contribute to better health. I and others responded with concern. The prospect is exciting but daunting. Current policies of regulating and paying for health goods and services reward introduction and overuse of expensive technologies that add incremental improvements in health, but with scant attention to cost or relative effectiveness. – Issues.org

This disconnect between spending and health outcomes is found in the following graphic.

The US spends the most on health care but underperforms the OECD average in health outcomes. Why doesn’t all of the spending at the NIH drive better health outcomes? 

This is expressed in the following quotation.

Between 80% and 90% of lung cancers have been linked to smoking tobacco. Yet of the $2.45 billion that the NIH has spent on trying to find a cure during the past decade, most has been directed towards the discovery of molecular and genetic causes and treatments rather than on establishing how to modify people’s behaviour.

In the United States, medical costs related to obesity (currently around $160 billion a year) are projected to double within the decade. And by some estimates, indirect spending associated with obesity by individuals, employers and insurance payers — for example on absenteeism, decreased productivity or short-term disability, exceeds direct medical costs by nearly threefold.

A second institute should be devoted to research on health outcomes, that is, on measurable improvements to people’s health. This should draw on behavioural sciences, economics, technology, communications and education as well as on fundamental biomedical research.  – Nature

The NIH Problems With Conflicts of Interest

This month, National Institutes of Health Director Francis Collins seemed to shut down a noxious ethical problem. The agency released a 165-page internal investigation of an alcohol consumption study that had been funded mostly by beer and liquor companies. The study’s lead investigator and NIH officials were in frequent contact with the alcohol industry while designing the study, which, according to the postmortem, seemed predetermined to find alcohol’s benefits but not potential harms, such as cancer.

This is the second time this year that Collins has been forced to distance his agency from corporate funding. In April, the NIH announced that it would decline money from drugmakers to support a proposed $400 million research program to discover opioid alternatives or new medicines to treat addiction.

Grassley had become interested in the topic in the 2000s after investigating Merck’s drug Vioxx — a painkiller tied to tens of thousands of heart attacks — and discovering that the Food and Drug Administration panel that approved it had been stacked with physicians with links to pharmaceutical companies. After the drug was removed from the market, the Journal of the American Medical Association published a study noting that Merck transformed peer-reviewed research into marketing brochures by ghostwriting studies for academics.

When President Barack Obama picked Collins in 2009 to run the NIH, Collins met with me and two other committee staffers as part of his nomination process. But whenever I pressed him about the ethics matters we were investigating, he shifted the conversation back to all the great science he wanted to do if the Senate confirmed him. (Collins says that he has always taken conflicts of interest seriously and that the agency upholds high standards for publicly funded research.) – Washington Post