Skip navigation

Bibliometrics

The CPI bibliometrics team is Australia’s leading centre for the systematic evaluation and mapping of research across all fields of scholarship. A major focus of the bibliometrics team is research on the advanced quantitative analysis of scientific performance and the organisational structure of Australia’s research landscape, and it regularly conducts bibliometric analyses of scientific publications produced under the patronage of publicly-funded bodies such as the ARC, CSIRO and NHMRC.

Parallel to this, the bibliometrics team also recognises the value of developing novel qualitative and quantitative approaches to research assessment, and the need to generate ‘metrics’ or indicators sensitive to the research and dissemination practices of a variety of fields not well served by standard bibliometric approaches, particularly in the social sciences, humanities and arts, as well as disciplines in the applied sciences such as computing and engineering.

With an increasing reliance on the use of quantitative measures in the assessment of research, members of the team play a central role in policy developments both within Australia, and internationally.

Database

The bibliometrics team has established a database containing all publications listing at least one Australian address that appear in journals in Institute of Scientific Information’s four main indices, the Science Citation Index (SCI), the Social Science Citation Index (SSCI), the Arts & Humanities Citation Index (A&HCI) and Current Contents (CC).

The following information is contained in the database for each publication:

  • Unique ISI publication code
  • Code for single author or type of collaboration: e.g. international, national
  • Name of journal
  • Publication year
  • Tape year (the year the publication was entered into the ISI database)
  • Number of addresses listed
  • Number of collaborating authors
  • Number of pages
  • Code showing sector (eg academic, government, etc) and ASRC field to which the address(es) given for the publication was classified
  • Publication type ie article, note, letter, etc
  • ISI Index in which publication is found
  • Standardised address(es)
  • Citations received by the publication in the year of publication and in each subsequent year

When studying the university sector, we are particularly interested in being able to undertake analysis at the level of department, and in order to do this we ‘clean’ the raw files obtained from ISI. We ensure that all variations of the same address are grouped together and that all publications with variants of that address are allocated to one ‘standardised’ address. For all universities and the CSIRO, addresses are ‘cleaned’ down to the department level, while in other sectors (with some minor variations) ‘cleaning’ is only done at the institution level.

This database is not restricted to university publications; it also covers hospitals, medical research institutes, government departments and research in the private sector. The database currently contains the details of 439,515 Australian publications covering the period 1981–2007. In addition to the publications data, information is also included on the number of times each of these items was cited by other publications in the ISI indices during the same period. This information is updated annually.

Past research projects

Project Retrosight: Returns on Investment in Cardiovascular Research

Project Retrosight is a three-year multinational study using a framework developed by the team, the Payback Framework, to investigate the impact of cardiovascular research, including how it is translated into clinical practice and ultimately how it effects health. The impacts of biomedical research occur over long time-scales and may take tortuous routes. The Payback Framework was originally designed by researchers at the Health Economics Research Group (HERG) at Brunel University to examine the benefits from health services research. Subsequently its scope was expanded to provide a way of handling the complexity of assessing the impact made by a wide range of biomedical research. The foundations of this work were a series of studies initially supported by the London NHS R&D Office. These were further developed through a number of studies led by RAND Europe, and conducted in collaboration with HERG, notably ‘The Returns from Arthritis Research’, a study for the UK’s Arthritis Research Campaign.

Researchers involved: L Butler and K Henadeera.
Researcher collaborators: Martin Buxton and Steve Hanney (HERG), Jonathan Grant and Steve Wooding (RAND Europe).
Project partners: Heart and Stroke Foundation of Canada, National Heart Foundation of Australia.

Development of measures of impact and quality for the RQF

REPP’s expertise on performance measures aimed at assessing the quality and impact of Australian university research across the full spectrum of disciplines was acknowledged in the appointment in 2006 of Ms Linda Butler (Metrics) and Dr Claire Donovan (Impact) to chair Department of Education Science and Training (DEST) Working Groups. These Groups were established to develop advice for the Minister on the design of the Research Quality Framework (RQF). Ms Butler continued to play a central role in developments throughout 2007, undertaking further empirical research into issues that remain unresolved. The REPP team has been coordinating the construction of discipline-specific journal rankings, and has been conducting pilot tests to test the design of proposed citation metrics.

Researchers involved: L Butler and C Donovan.

Strategic Assessment of Research Performance Indicators (ARC Linkage Project)

The use of quantitative measures of research performance in higher education sends powerful messages to those being evaluated, because the choice of measures signals what is considered most important. However, little critical assessment has been undertaken on the use of performance indicators or their impact on research practice. This project will create a knowledge base on performance measures - their validity, fairness, transparency and impact on research, and the cost of implementation. By assessing an extensive range of measures, it will provide Australian science management and science policy makers with rigorous information on which to base informed judgements on their utilisation.

Researchers involved: L Butler, G Laudel and C Donovan.
Researcher collaborators: Frank Jackson (Philosophy, RSSS), David Siddle (University of Queensland) and Ian Lucas (Department of Education, Science and Training).

Links to further details:

Impact of Evaluation-Based Funding on the Content of Scientific Research (ARC Discovery Project, additional funding from BMBF, Germany)

Evidence suggests that evaluation-based funding mechanisms trigger adaptive behaviour in scientists, leading to unintended changes in knowledge production. However, there is no conclusive evidence about when (under what circumstances), how (by what processes) and with what effects researchers adapt to these mechanisms. This study aims to answer these crucial questions. To overcome the methodological weaknesses of earlier studies, a comparative investigation of several research fields and universities will be conducted, combining for the first time: qualitative interviews, ethnographic observations and bibliometric studies. It will enhance Australian research policy with a critical assessment of existing research funding models.

Researchers involved: J Gläser, G Laudel, L Butler
Collaborators: Uwe Schimank, Fern Universität in Hagen, Germany
Further information: 2005_discovery_grant.html

Updated:  22 February 2011/Responsible Officer:  Professor of Public Policy, CPI /Page Contact:  Professor of Public Policy, CPI