Bibliometrics is the quantitative analysis of publications and citations. It is used to evaluate researchers, journals and institutions, and to map how research fields develop. Few numbers have shaped academic careers as much as the journal impact factor and the h-index. Both are widely misused.
This guide explains how the main metrics work, what they measure and what they do not, and how evaluation practice is changing.
The main metrics
| Metric | How it is calculated | What it measures |
|---|---|---|
| Journal Impact Factor (JIF) | Citations in year Y to items a journal published in Y−1 and Y−2, divided by citable items published in those two years | Average recent citation rate of a journal |
| CiteScore | Like the JIF but over a four-year window and all document types (Scopus) | Same idea, different database and window |
| h-index | A researcher has index h if h of their papers have at least h citations each | Combined productivity and citation impact of a person |
| Field-normalised citation impact | Citations of a paper divided by the average for papers of the same field, year and type | Impact relative to comparable work |
| Top-percentile share | Share of a unit’s papers among the 1% or 10% most cited in their field | Excellence, less sensitive to outliers |
| Altmetrics | Mentions in news, policy documents, social media and reference managers | Attention outside academic citation |
What the numbers do not tell you
The impact factor describes journals, not papers. Citation distributions are highly skewed. A small share of papers receive most citations, so most papers in a high-impact journal are cited less than the journal’s average. Judging a paper or a researcher by the JIF of the journal where they published is a category error.
Citation practices differ by field. A typical biomedical paper is cited far more than a typical mathematics or humanities paper. Comparing raw citation counts across fields rewards the field, not the researcher. Use field-normalised indicators.
The h-index favours seniority. It can only grow with time, it ignores how a researcher’s contribution compares with co-authors, and it treats a paper with 50 citations the same as one with 5,000 once both exceed h. Jorge Hirsch proposed it as a simple summary, not as a hiring rule.
Metrics can be gamed. Self-citation, citation cartels between journals, coercive citation by editors, salami-slicing results into many small papers and paper mills all inflate numbers. Any metric used for high-stakes decisions becomes a target. This is Goodhart’s law.
Databases differ. Web of Science, Scopus, Google Scholar, Dimensions and OpenAlex cover different journals, languages and document types. Non-English research, books and regional journals are under-represented in the traditional databases. That matters a great deal for Chinese-language research, for example.
Responsible research assessment
Concern about misuse has produced a series of influential statements:
- DORA (2012), the San Francisco Declaration on Research Assessment, recommends not using journal-based metrics such as the impact factor to assess individual researchers, articles or funding decisions. It has been signed by thousands of organisations.
- The Leiden Manifesto (2015) sets out ten principles. Quantitative evaluation should support, not replace, expert judgement. Measure performance against the institution’s own mission. Keep data and analysis transparent. Account for field differences. Review indicators regularly.
- CoARA (2022), a coalition of European funders, universities and academies, commits signatories to assess research mainly through qualitative peer review, supported by responsible use of indicators.
China moved in a similar direction in 2020, when the education and science ministries issued guidance against “SCI-only” evaluation. Institutions were told to stop using Science Citation Index paper counts and impact factors as decisive criteria for hiring, promotion and funding. The guidance was part of a wider campaign against the “five onlys” (破五唯): judging people only by papers, titles, diplomas, awards or academic honours.
Mapping research fields
Bibliometrics is at its best not for ranking people but for understanding fields:
- Co-citation analysis. Papers frequently cited together are probably related. Clusters of co-cited papers reveal a field’s intellectual structure.
- Bibliographic coupling. Papers that cite the same sources are probably working on similar problems. This is useful for mapping current research fronts.
- Co-word analysis. Keywords that appear together reveal themes and how they shift over time.
- Co-authorship networks. Reveal collaboration patterns between people, institutions and countries.
Free tools have made these methods widely accessible. VOSviewer and CiteSpace build and visualise networks, and OpenAlex provides an open, free database covering hundreds of millions of works. A typical workflow: define the field with a search query, export records, clean author and institution names, build maps, and then read the key papers in each cluster to interpret what the map shows. The last step is the one most often skipped, and the one that matters most.
Bibliometrics and AI
Two changes are under way. Open databases and AI-assisted classification make large-scale analysis cheaper and less dependent on commercial providers. Meanwhile, generative AI is raising new integrity questions, from fabricated references to machine-generated papers, which makes careful verification of the literature more important. Used responsibly, metrics remain useful for spotting patterns that no reader could see alone. They work as a map, not as a verdict.
Sources
- J. E. Hirsch: An index to quantify an individual's scientific research output (PNAS, 2005)
- San Francisco Declaration on Research Assessment (DORA)
- Hicks, Wouters, Waltman, de Rijcke & Rafols: The Leiden Manifesto for research metrics (Nature, 2015)
- Coalition for Advancing Research Assessment (CoARA)
- OpenAlex: open catalogue of the global research system
- VOSviewer: software for constructing and visualising bibliometric networks