What makes a good OpenPrescribing measure?

How do we decide which measures to include at OpenPrescribing?

This blog describes how we develop new measures at OpenPrescribing. We will describe the steps in the measure development process including the review of technical aspects and how we think about the impact and interpretation of a new measure.

Where do we get ideas for measures?

Most ideas for new measures come from new guidelines or policy updates. The most recent example of this was the NHS England National medicines optimisation opportunities 2023/24 guidance and the recent update to the Items which should not routinely be prescribed in primary care.

We also build measures based on drug safety updates from the MHRA (for example, for prescribing ciclosporin and tacrolimus by brand), inspiration from our users (for example our high dose opioid items as percentage regular opioids measure), and from within our own team of clinical informaticians who spend half their time in practice!

What is the process?

We spend a lot of time scoping, prototyping and testing measures before they are released openly onto the website. We first decide amongst our team of Clinical Informaticians whether a measure should be built at all and then one of us goes ahead and builds a prototype. When a prototype has been built, it is shared amongst the team for review. We can also share the prototype with interested users, to get their feedback on the design of the measure and its use.

What do we review?

Technical aspects

Is the measure valid?

We build our measures using the programming language SQL, so the first check is whether our coding is correct. Are we using the right measurement e.g. Quantity vs Items? For example, if you wanted to measure how many days supply of antibiotics were supplied then we would use Quantity eg. Antibiotic stewardship: courses for amoxicillin 500mg greater than 15 capsules and if we wanted to look at the number of times a medicine has been prescribed we would use Items eg. Diltiazem preparations (>60mg) prescribed generically. More on this in our FAQs.

Numerator and denominator

Are all medications included within the numerator and denominator that we expect? Are we using an appropriate denominator e.g. practice list size, STAR-PU (more on this here) or group of medications.

Have we followed OpenPrescribing conventions?

We try to standardise the interpretation of our measures with a “low is good” approach. In almost all of our measures, a lower score indicates a better outcome. For example, that users are following best practice guidelines or using the most cost effective option.

Is the SQL future proof?

In one of our recent blogs we describe some approaches to SQL that enable us to pick up new medications or formulations without having to keep a fully up-to-date list.

Impact and interpretation

Have we labelled the measure clearly?

At OpenPrescribing we want users to have clarity on exactly what is being presented in each measure. The measure should be easy to interpret. The composition of the numerator and denominator and whether we are reporting items or quantity (or any other analytic choice) should be clearly displayed and be easily reproducible.

Is the measure useful?

We want measures to highlight key areas of prescribing where improvements will yield a large impact to cost-savings, safety, efficacy, or sustainability. We want to draw attention to prescribing patterns of high importance rather than distract with measures that are not likely to improve practice.

Could the measure be misinterpreted?

A simple numerator denominator measure in OpenPrescribing is often not suited to describing some prescribing patterns. We want measures to reflect, in the majority of cases, a straightforward prescribing decision where there is a clear ‘right answer’. Where it is unclear whether higher or lower reflects better performance, we think carefully about whether there is still value in showing the measure, e.g. there may still be value in allowing organisations to see if they are an outlier.

Is the measure meaningful without indication / patient characteristics?

In OpenPrescribing we cannot link our data to any patient-level data such as indication, age or sex. This is taken into consideration when developing measures in case a measure is meaningless without this information.

Two recent examples

We recently developed two measures for adrenaline auto-injectors to reflect MHRA guidance. The two aspects of the MHRA guidance that we wanted to report were the percentage of adrenaline auto-injectors not prescribed by brand and a quantity of one adrenaline auto-injector on a prescription. These two measures were built as prototypes and ultimately were rejected. The reasons why are described below.

The percentage of adrenaline auto-injectors not prescribed by brand

After technical review we set to work thinking about the impact and interpretation of this measure. From the prototype we saw that, in England, the vast majority of adrenaline auto-injectors were prescribed by brand.

Preview: Percentage of adrenaline auto-injectors not prescribed by brand. OpenPrescribing.net, Bennett Institute for Applied Data Science, University of Oxford, 2023

When we looked at the plots, the proportion of adrenaline auto-injectors not prescribed by brand was 0.789% in August 2023 and the 90th percentile <2%. The overall number of generic auto-injectors prescribed nationally per month was low (264/33475). If we were to extract the lowest 10% we could be highlighting practices achieving >97% appropriate prescribing. In most scenarios, 97% is a very good score, and not relevant to highlight to busy clinicians or medicines optimisation teams.

Since the numbers were low and prescribing was mostly by brand this measure was dropped as we felt it could distract from other more impactful measures.

Quantity of one adrenaline auto-injector on a prescription

After technical review, we discussed the interpretation of this measure. This measure suggests that a prescription for a single adrenaline auto-injector should be avoided. However, this could be a very careful prescriber, who has discussed how many adrenaline auto-injectors the patient needs to maintain a supply of two and is not being wasteful.

We then tested a quantity of >2 adrenaline auto-injectors on a prescription. This led to further discussion around misinterpretation. In some scenarios, as described here, healthcare professionals are advised to prescribe more than two, so a spare pen is available at school for example.

In short, the quantity of adrenaline auto-injectors cannot be described as an OpenPrescribing measure because there are a number of different ‘correct answers’.

Summary

At OpenPrescribing, we receive lots of ideas for new measures, but not all ideas make it onto the platform. Here we summarise the main attributes of a good measure!

A good measure:

• Has clear cost saving, green, safety or efficacy benefits to OpenPrescribing platform users and patients!
• In most cases has a single correct answer (unlike the number of adrenaline auto-injectors to supply!)
• Is not dependent on any demographic detail or the reason for the medication
• Should show some clear widespread room for improvement at the time of development
• Has clearly written, reproducible, future proof SQL
• Is clearly labelled and easy to interpret

Please get in touch!

If you have any comments or suggestions for any part of OpenPrescribing please do not hesitate to get in touch with us, either by email or tagging us on X (formerly Twitter).