Introduction
Deriving a fair, unbiased, and easily generated quantitative index
serving as a reasonable first-pass metric for comparing the relative
performance of academic researchers is — by the very complexity,
diversity, and intangibility of research output across academic
disciplines — impossible (1). However, that unachievable aim has not
discouraged bibliometricians and non-bibliometricians alike from
developing scores of citation-based variants (2, 3) in an attempt to do
exactly that, from the better-known h-index (4, 5) (h papers with at least h citations), m-quotient (4, 5)
(h-index ÷ number of years publishing), and g-index (6)
(unique largest number such that the top g papers decreasingly
ordered by citations have least g2 citations),
to the scores of variants of these and other indices — e.g., h2-index, e-index (7), χ-index
(8), hm-index (9), gm-index (10), etc. (3). Each metric has its own
biases and strengths (11-13), suggesting that several should be used
simultaneously to assess citation performance. For example, the arguably
most-popular h-index down-weights quality relative to quantity
(14), ignores the majority of accumulated citations in the most highly
cited papers (15), has markedly different distributions among
disciplines (16), and tends to increase with experience (17). The h-index can even rise following the death of the researcher,
because the h-index can never decline (2) and citations can
continue to accumulate posthumously.
Despite their broad use in everything from assessing candidates applying
for academic positions, comparing the track records of researchers
applying for grants, to applications for promotion (3, 18) inter
alia, single-value citation metrics are rarely meant to (nor should
they) be definitive assessment tools (3). Instead, their most valuable
(and fair) application is to provide a quick ‘first pass’ to rank a
sample of researchers, followed by more detailed assessment of
publication quality, experience, grant successes, mentorship,
collegiality and all the other characteristics that make a researcher
more or less competitive for rare positions and grant monies. But
despite the many different metrics available and arguable improvements
that have been proposed since 2005 when the h-index was first
developed (4, 5), few are used regularly in these regards. This is
because they are difficult to calculate without detailed data of a
candidate’s publication history, they are not readily available on
open-access websites, and/or they tend to be highly correlated with the h-index anyway (19). It is for these reasons that the admittedly
flawed (20, 21) h-index and its experienced-corrected variant,
the m-quotient, are still the dominant (h-index much more
so than the m-quotient) (2) metrics employed given that they are
easily calculated (2, 22) and found for most researchers on open-access
websites such as Google Scholar (23) (scholar.google.com). The lack of
access and detailed understanding of the many other citation-based
metrics means that most of them go unused (3), and are essentially
valueless for everyday applications of researcher assessment.
The specific weaknesses of the h-index or m-quotient make
the comparison of researchers in different career stages, genders, and
disciplines unfair because they are not normalised in any way.
Furthermore, there is no quantitatively supported threshold above or
below which assessors can easily ascertain minimum citation performance
for particular applications — while assessors certainly use subjective
‘rules of thumb’, a more objective approach is preferable. For this
reason, an ideal citation-based metric should only be considered as a
relative index of performance, but relative to what, and to whom?
To address these issues and to provide assessors with an easy, rapid,
yet objective relative index of citation performance for any
group of researchers, we designed a new index we call the
‘ε-index’ (the ‘ε ’ signifies the use of residuals, or
deviance from a trend) that is simple to construct, can be standardised
across disciplines, is meaningful only as a relative index for a
particular sample of researchers, can be corrected for career breaks
(see Methods), and provides a sample-specific threshold above and below
which assessors can determine whether individual performance is greater
or less than that expected relative to the other researchers in the
specific sample.
With the R code we provide, an assessor need only acquire four separate
items of information from Google Scholar (or if they have access, from
other databases such as Scopus — scopus.com) to calculate a
researcher’s ε-index: (i) the number of citations
acquired for the researcher’s top-cited paper (i.e., the first entry in
the Google Scholar profile), (ii) the i10-index
(number of articles with at least 10 citations), (iii)
the h-index, and (iv) the year in which the
researcher’s first peer-reviewed paper was published. While the last
item requires sorting a researcher’s outputs by year and scrolling to
the earliest paper, this is not a time-consuming process. We demonstrate
the performance of the ε-index using Google Scholar citation data
we collected for 480 researchers in eight separate disciplines spread
equally across genders and career stages to show how the index
performs relative to the m-quotient (the only other readily
available, opportunity-corrected citation index available on Google
Scholar) across disciplines, career stages, and genders. We also provide
a simple method to scale the index across disciplines (ε′-index)
to make researchers in different areas comparable despite variable
citation trends within their respective areas.