The Genius Index: One Scientist's Crusade to Rewrite Reputation Rules

Jorge Hirsch had been getting screwed

In the scientific community, journal citations are the coin of the realm. *
Illustration: Ronald Kurniawan * Jorge Hirsch had been getting screwed. For years. At a scientific conference in 1989, he presented a paper arguing that the generally accepted theory of low-temperature superconductors—the BCS theory—was wrong. Most researchers at the time held that under certain low-temperature conditions, vibrations in a metal's crystal lattice can allow electrons to become attracted to one another, which drops electrical resistance to zero—a superconducting state. Hirsch said this "electron-phonon interaction" in fact had nothing to do with superconductivity. He was a youngish up-and-comer then, but physics rarely forgives apostasy. After his fateful presentation, similar conferences stopped inviting him to speak. Colleagues no longer sought him out for collaboration. Grants dried up. High-visibility journals shunned his papers.

It's not that Hirsch wasn't getting his work published. He was. And other physicists were still citing his research, implying some acceptance of his views. Hirsch just wasn't able to get his papers into the really high-visibility journals—places like Science, Nature, and, for a solid-state physicist, Physical Review Letters. There's a clear pecking order, established and reinforced by several independent rating systems. Chief among them: the Journal Impact Factor.

Hirsch, like his peers, understood that if he wanted to get to the front ranks of his discipline, he had to publish in journals with higher JIFs. But this struck him as unfair. Like most physicists, he's pretty smart, and not shy about it. The system that had shunned him was wrong, Hirsch concluded. It shouldn't be about where he published; it should be about his work. So in 2003, Hirsch decided that he—oh, and science, too!—needed a system that would rate researchers directly.

After two years of number-crunching in his cluttered office at UC San Diego, Hirsch had it—an invention important enough to warrant publication in the (very prestigious) Proceedings of the National Academy of Sciences. In his 2005 article, Hirsch introduced the h-index (named after himself, of course). The key was focusing not on where you published but on how many times other researchers cited your work. In practice, you take all the papers you've published and rank them by how many times each has been cited. Say paper number one has been cited 10,000 times. Paper number two, 8,000 cites. Paper number 32 has 33 citations, but number 33 has received just 28. You've published 32 papers with more than 32 citations—your h-index is 32.

Or to put it more technically, the h-index is the number n of a researcher's papers that have been cited by other papers at least n times. High numbers = important science = important scientist.

In its nearly four years of life, the relatively simple, flexible h-index has become the most talked-about metric in the very hot science of rating scientists and their research, a discipline that has flourished in ways Hirsch had never imagined. The h-index was the biggest splash in a flood of Internet-enabled rating systems—growth and decay chronometrics, semiometric measures, hub/authority metrics. Schools and labs use such ratings to help them make grants, bestow tenure, award bonuses, and hire postdocs. In fact, similar statistical approaches have become standard practice in Internet search algorithms and on social networking sites. These numbers do for scientists what U.S. News & World Report does for colleges and Bill James' Sabermetrics did for baseball: They quantify reputation.

Or at least they try. Either way, the numbers have the stark, uncluttered feel that scientists love. "I'm a physicist," Hirsch says. "Some people put a lot of weight on subjective criteria, and I don't like that at all."

The science citation revolution began more than 50 years ago. Eugene Garfield, then a young librarian pursuing a PhD in structural linguistics, started wondering about that most prosaic of bibliographic tools: the footnote. Most people think of footnotes as reaching backward in time to a document's sources. But Garfield realized that they could reach forward, too—future footnotes would cite the original article. "The citation becomes the subject," says Garfield, now 83 and enjoying his stature as the founding father of modern citation analysis. "It was a radical approach to retrieving information."

Some three decades before the concept of the hyperlink and the World Wide Web crossed anybody's mind, Garfield had figured out how to connect the immense body of scientific knowledge into a network. In the early 1960s he began publishing The Science Citation Index; Garfield sold the first edition, five volumes of arcane hard-copy reference, to academic libraries for $500.

Over the years, as the index expanded, Garfield worked furiously to figure out how to select and compare the journals it included. He found his answer in a simple equation: Divide the number of citations the journal gets over the prior two years by the number of articles it published. He named it the Journal Impact Factor, and in 1975 he decided to reveal how it worked. Only librarians seemed to care. "It was clear that most people didn't have the vaguest idea what I was talking about," Garfield says.

Still, it was a tidy little business, and in 1992 Garfield sold it to what's now called Thomson Reuters. But it didn't get much traction until five years later, when the company dumped the entire database onto the Internet. Today, it's part of the ISI Web of Knowledge, and its arrival online gave scientists easy access to a colleague's oeuvre so they could see whether it was published in good journals. Suddenly, the scientific life had a scoreboard.

The Web of Knowledge now comprises 700 million cited references from 23,000 journals published since 1804. It's used by 20 million researchers in nearly 100 countries. Anyone—scientist, dean, lab director—can sort the entries and tell someone's fortune. Nothing approaches it for breadth and longevity. Though the Journal Impact Factor has competitors, it remains the gold standard. "You may not like the database, but it has not been replaced," Garfield says.

That said, the JIF has problems. Review journals, which round up current research, are generally cited far more often than those that publish new findings. And even though the JIF is meant to rate only journals, most people use it as a proxy for rating individual scientists anyway.

Worse, the system can be gamed. Editors have been known to encourage authors to cite articles from their journal (which has the aroma of extortion—"Help us with our JIF if you want to get published"). Journals that print obituaries, editorials, and letters catch a break, because these articles are included in the JIF numerator but not counted in the denominator. There had to be a better way.

Even change-averse academia admitted that Hirsch had come up with a different mousetrap. "The h-index does seem to be able to identify good scientists, and it is becoming widely used informally," the journal Nature reported in 2007, perhaps somewhat grudgingly. The index makes it harder to cheat. Researchers have to be champion self-citers to move their own numbers, and editors have no reason to influence the system.

On the other hand, Hirsch acknowledges that the h-index has its own intrinsic weaknesses. It's kind to older folks, for example, but not great to younger scientists. If a scientist writes six brilliant papers and dies, his h-index will never be higher than six, even if each paper is cited 10,000 times. And by putting the onus on individuals, it encourages researchers to write about sexy topics and hew close to the conventional wisdom—exactly what Hirsch was trying to avoid. Plus, it has trouble apportioning credit on papers with multiple authors. (Complicated math might sort researchers by their respective contributions ... maybe.)

While the problems of the various citation-based ranking schemes might seem (ahem) academic, their strategies are increasingly the coin of the online realm. Understanding and quantifying reputation is the best approach to navigating the tsunami of information on the Internet. That's why Google founders Larry Page and Sergey Brin cited Eugene Garfield in their academic work on PageRank, the algorithm that powers their company's search engine. "Articles cited by this article" and "articles that cite this article" are really just outbound and inbound links. Today, citation analysis has come full circle. Eigenfactor and SCImago actually use variations of PageRank to evaluate scientific journals. And the introduction of Google Scholar, a search tool designed specifically for academic research, provided a whole new set of citation data; it can help calculate h-index, as well as a newer ranking system called the g-index that gives more weight to articles with higher citation counts.

The depth of data available on the Internet and the increasing importance of online archives and publications like the ArXiv for physics and the Public Library of Science for biomedicine have made possible new metrics: number of downloads, rate of downloads over time (chronometrics), and even levels of funding and numbers of doctoral students working in a lab (nonbibliometric performance indicators). So powerful are these new kinds of measurements that they're finding applications beyond ranking scientists. One recent article suggested that citation analysis could be used to predict the direction of scientific innovation.

Not even Web flavor-of-the-month Twitter is immune. The microblogging site displays the number of users following a given Twitterer, and the number of users that person follows. So you can see where this is going: Various Web sites track Twitter use, factoring in those numbers as well as "re-tweets," the number of times a person's posts get recapitulated, to come up with ranks for users. The more important any given Web application becomes, the more its users want ranks.

"The Impact Factor was never supposed to be used to rank scientists," says Marie McVeigh, senior manager for bibliographic policy at Thomson Reuters. She's right, of course, but it's too late now. At least the h-index seems to give people the right numbers. Edward Witten, genius cosmologist at the Institute for Advanced Study, scores 120, the highest of all physicists. A Brief History of Time theorist Stephen Hawking gets 67. And Hirsch? He rates a 52, which according to his 2005 paper makes him "outstanding" and "likely to be found only at the top universities or major research laboratories." See? The system works.

Guy Gugliotta (guygugliotta@yahoo.com) wrote about mutated microorganisms in issue 15.09.

Highlights From the New Peer-Reviewed Creation Science Journal

Jan. 23, 1911: Science Academy Tells Marie Curie, 'Non'

Peer Production