Measuring Surgical Outcomes

The Royal College of Surgeons of England is working toward improved methods of ensuring high standards in surgical practice through public reporting of operation outcomes. A reliable system of measuring outcomes will have many benefits:

• Greater public transparency and accountability.
• Enable surgeons a better basis for judging and improving their practice.
• Offer patients the basis to make informed choices about their care.
• Evidence for service improvement and quality assurance of operations.
• Better data for health service commissioners when making funding decisions.

A sound system for outcome measurement will combine existing statistics and audits, new clinical registers, and patient attitudes to the results of their operation.

Barriers to outcome measurement

If national outcome measurement for surgery is so desirable – why don’t we already have a system? In short, because arriving at a meaningful, intelligible and fair system is extremely complicated.

The following issues need to be addressed:

• The difference between specialties – there are ten surgical specialties and the surgeons working in them all do very different and highly specialised work. So the methods of measuring outcomes will need to be tailored to each – there can be no “one size fits all” approach.
• What do we measure? - Deciding which operations define a surgeon’s ability and are done frequently enough by all surgeons within a specialty to get statistically meaningful data will be tougher for some areas than others.
• How do we measure? – What is considered a successful operation will very much depend on the type of procedure, the seriousness of the condition it is aiming to treat and the age/fitness/expectation of the patient.
• Timescale – How long after the operation does its true outcome become apparent? For many operations it may be some years afterwards, for others the results may be instantaneous.
• Patient reported and clinical analysis – Both are important and should be given due weight. But evidence-based, clinical analysis operates by a scientific set of measurements; whereas patient experience is far more nuanced and subjective. Any system of outcomes will need to be sophisticated enough to allow these two worlds to meet – and that will take refinement and experience.
• How do we publish? – With a lot of complex statistical data to be gathered and reconciled, careful planning must be put into how this data is published for various audiences – especially the general public for whom the information must be both easily understandable and sophisticated enough to give them valid choice.
• Risk adjustment – Statistics gathering must take account of surgeons who take on the highest risk cases or are developing new techniques. Outcome measurement must not encourage a spirit of conservatism in medical practice.
• Data quality – variation in the quality of how data is recorded exists across the NHS and improved data systems. Training for clerks and harmonisation of processes must be a priority for government if the system is to produce valid results.
• Team v individual data? – Modern medical practice is increasingly focused on working across disciplines and aiming for better care by inter-reliance between specialists. While being refined, outcome measures should focus on team results to ensure their validity before drilling down to individual level.

There already is some public data on operations – for example, Hospital Episode Statistics – but these measure procedure (eg. re-admission rate or length of stay) and can only provide an arbitrary measure of overall success. True outcome measurement will require a series of measures subtle enough to take account of the individual situation and satisfaction of each patient.

What progress has been made?

The Royal College of Surgeons and the surgical specialty associations are leading the way in developing methods for outcome measurement.

Cardiac surgeons already have a public database of their individual results and mortality rates for select procedures.

Their measurement system focuses only on heart bypass operations and, due to the high risk of this kind of surgery, has a very definite measure – whether the patient survives  Cardiac surgeons have some 30 years of experience in registering surgical outcomes and have refined their approach so that they can use the system to compare their performance against their colleagues and take steps to improve their skills wherever they can.

The College has led the way in piloting a system of patient reported outcome measures (PROMs), initially to audit the performance of independent sector treatment centres. The Department of Health is funding an extension of the approach across the whole NHS and, from 2008, all patients undergoing hip, joint, hernia and varicose veins will be covered and this will account for 250,000 operations a year. This work will combine with the work the college is doing in refining surgeon reported outcomes to give a more holistic picture of what the best surgery can deliver for patients.

The surgical specialties are now identifying the “indicator” operations that would give a sound judgement of skill and are done by enough of their surgeons to be good comparators. Some, such as orthopaedic and general surgery, also have past experience in registering outcome and are consolidating this material. This puts them in a strong position to follow the cardiac surgeons lead.