This blog describes some significant methodological concerns and misrepresentation of data within research articles and reports written, published and/or funded by Oxehealth. We aim to describe these issues as simply and as clearly as possible for readers who aren’t familiar with statistics (though this may feel too simple for people who are). In addition to writing this blog, we have written to the editors of each of the academic journals where the articles in question are published. We have been reassured that their respective ethics and research integrity teams are currently looking into these issues.
Please note that this article includes extensive discussion of self-harm. This includes, at one point, mention of specific injuries associated with self-harm.
Research related to Oxevision
Oxehealth have funded several research studies which look at the implementation of Oxevision in mental health wards across England. These have considered the cost outcomes, patient views and other measurements such as the number of incidents of self-harm or falls. Concerns have been raised about the quality of this research. In one case, an article was retracted due to the authors’ failure to declare conflicts of interest. Additionally, a systematic review (an in depth review of previously published research) identified the poor quality of research looking at Oxevision.
Further concerns have been raised about the ethics of research related to Oxevision. For example, a team of researchers at London South Bank University conducted a study which involved viewing footage of patients in their bedroom. The video footage was obtained through Oxevision at three NHS sites (EPUT, RDaSH and TEWV). Patients did not consent to take part in the study and were not aware that videos of them were used for research. As the video footage was blurred, the researchers argued that this was anonymous, however, following a complaint written by Stop Oxevision (among others), the ethical approval for this study was retracted.
Amongst these issues with the research related to Oxevision – and in addition to the ethical concerns with the technology itself – this blog will describe some specific errors made in some of the research Oxevision have conducted or commissioned. We believe these errors are significant, and are concerned that the incorrect statistics have been used by Oxehealth in many of their promotional materials. We will discuss two main issues:
- Statistics are presented in a way which confuses a relative change, in comparison to a ‘control ward’ with an actual reduction. (We will give a much clearer description of this later on).
- An economic analysis has been calculated based on a) misinterpreted statistics and b) illogical/incorrect cost data, and as a result inflating the figures they report as being the ‘cost savings’ of implementing Oxevision.
These issues relate to data which has been repeated in multiple academic journals and publications on Oxehealth’s website.
The publications our concerns relate to
Oxehealth use the incorrect figures in a number of places across numerous promotional materials. In particular, our concerns relate to three peer-reviewed academic publications (meaning that in order for them to be published they are reviewed by 2+ expert reviewers and an editorial team). These are as follows:
- Non-contact monitoring to support care in acute inpatient mental health, published in Journal of mental health
- Economic impact of a vision-based patient monitoring system across five NHS mental health trusts, published in PLOS digital
- Economic evaluation of a vision-based patient monitoring and management system in an acute adult and an older adult mental health hospital in England, published in the Journal of Medical Economics
Two of these publications relate to an economic analysis which was commissioned by Oxehealth. Oxehealth commissioned a York Health Economics Consortium to conduct an economic analysis which looks at how much money mental health trusts could save by implementing Oxevision. The health economists concluded that:
“the results indicate that, for an average ward in an NHS mental health trust, Oxevision could produce an annual incremental benefit value of £52,646 for older adult services, £69,181 for PICU and £37,523 for acute services over one year”.
Health economics is focused purely on the financial cost of health services. The cost of a treatment or intervention is compared to the possible cost savings it could have. This could be through saving staff time or reducing the length of time in hospital. In this analysis, the cost of Oxevision is compared to possible cost savings in terms of staff time, and the cost saved by the supposed reduction in incidents such as “self-harm, falls, assaults and restraints”.
The other publication was conducted by Oxehealth and Coventry and Warwickshire NHS Trust. This compared a range of measurements before and after implementing Oxevision across adult acute, PICU and older adult mental health wards. Some of the data from this research, specifically that relating to rates of self-harm on the wards, are misrepresented in a number of places.
Does Oxevision lead to a reduction in self-harm?
Oxehealth and Coventry and Warwickshire NHS Trust conducted research where they looked at data for the wards with Oxevision and compared it to wards without Oxevision. They collected 12 months of data from January to December 2019. This was compared to the existing data they had for the previous year (where Oxevision had not been used).
Here is a report from Coventry and Warwickshire:
A few notes:
- Self-harm was measured through Datix incidents (an NHS incident reporting system) which were listed as self-harm.
- The numerous publications also do not address the important distinction between the actual rate of self-harm and the rate that is observed and documented by staff in a way which then shows up as part of this data. Self-harm takes many forms and much of this will not be observed and/or recorded by staff members. Therefore, there will always be some difference between what really happened and what the data shows. Additionally, multiple separate occurrences of self-harm may be reported within one incident form.
- ‘Correlation does not mean causation’ – a well used phrase in research – emphases that although there is a relationship between variables (in this case having Oxevision and the number of self-harm incidents) that doesn’t mean one thing necessarily caused the other. There are many other factors which could influence this.
What does a 22% reduction in self-harm mean?
This research shows that during the 12 month period, on the ward which had Oxevision, there were 22% fewer reported incidents of self-harm than there were in the previous 12 months (before Oxevision was introduced).
This figure was not statistically significant. In very simple terms, a figure which is statistically significant means that researchers can be confident that the results would not have occurred at random. If a figure does not reach statistical significance then a researcher would not be able to confidently conclude that there was any effect identified.
The authors report a 95% confidence interval between -100% and +19%. Again, in very simple terms, this is the range in which the researcher can be confident the true value lies. This is a wide confidence interval and includes values which would be an increase and a decrease in the amount of self-harm. That shows that there was quite a lot of variation in the number of reported incidents of self-harm in patient bedrooms/en suite bathrooms.
However, while there were 22% fewer reported incidents of self-harm on the control ward during the Oxevision year, there was a large increase in the number of self-harm incidents on the control ward. During the 12 month period, on the ward which did not have Oxevision there were 39% more reported incidents of self-harm than there were in the previous 12 months. That is a fairly big change, and is bigger than the change was on the ward with Oxevision.
What is a ‘control ward’?
In health research a control group is often used. The purpose of this is to be able to compare a group of people who have received an intervention to a group of people who have not. For example, if you were testing the effects of a blood pressure tablet, you might give this to group A and give group B a fake sugar pill (placebo). Having the control group would allow you to see whether the change in blood pressure was because of the tablet, or what is known as a placebo effect (or other influences). If both groups’ blood pressure reduces by the same amount, even though only one group were taking the real tablet, you can’t be sure that the actual medication was doing anything.
Sometimes an intervention group is compared to what is known as a ‘treatment as usual’ group, who receive the care that is usually given. This allows researchers to see whether their new treatment is better than what patients are currently offered.
In the case of Oxevision’s research, the ‘control ward’ would’ve given the same treatment as usual – in person observations, no monitoring devices. In this case, having the data for this ward allows Oxehealth to see whether the rate of reported self-harm incidents changed and to be able to compare this with the ward which had Oxevision.
But the most notable finding from their research was that on the wards where patients were receiving usual treatment, there was more self-harm in 2019 than there was in 2018. However, if the control ward experienced the same conditions as usual then there is no clear reason why the levels of self-harm on this ward changed so much. At least, the authors (and Oxehealth) have never discussed this or addressed this key limitation. There are a few possibilities for this:
- The number of self-harm incidents reported just varies hugely from year to year because it depends so much on the patients on the ward at the time.
- There was a push to make sure staff are appropriately reporting incidents leading to what appears to be an increase but is actually just more accurate reporting.
In clinical trials there is something known as blinding. This refers to where a researcher doesn’t know which group (treatment or control) a participant is allocated to. This is generally thought to improve the quality of research as there is a risk that if the researcher knows which group a patient is in, then they could accidentally (or deliberately) influence the result. Similarly, this also prevents researchers from specifically putting patients into different groups to inadvertently (or deliberately) manipulate the results.
It is not always possible to ‘blind’ participants or researchers to which intervention a patient is receiving or how they are allocated to a group. For example, in the case of this research, patients would be admitted to wards as there was space.
However, whilst we are not implying that this was the case, it should be noted that one possibility for the increase in self harm on one ward and decrease on the other, was that knowledge of the research may have subconsciously influenced which ward patients were admitted to. Logically, one possible way to reduce self-harm on a ward is to move all the patients who self-harm to the other ward, for example.
Ultimately, as the ‘control wards’ had so much variation in the number of incidents of self-harm between the two years, this research can’t really show us much other than that the rates of reported incidents of self-harm can change a lot, even when you don’t change anything about the ward. Therefore, saying that the Oxevision ward had 22% fewer incidents of self-harm misses out a lot of the important points about what the research really showed.
The difference between an absolute and a relative change
The 22% reduction previously discussed is what is called an absolute change. This means it is the actual change in the number of incidents of self-harm from 2018 to 2019. A relative change, on the other hand, expresses (in this case) the change as the amount that the number of reported self-harm incidents on the oxevision ward changed between 2018 and 2019 in relation to the amount that it changed on the non-Oxevision ward. This is calculated by dividing one percentage change by the other. Basically, -22% divided by 39% (then minus one and multiply by 100 to express as a percentage) is – 44%. That means that the amount that reported incidents changed on the Oxevision was 44% less than the amount that self-harm changed on the non-oxevision ward. The main point to take away is that this does not mean that the number of self-harm incidents decreased by that amount on the Oxevision ward.
This study also reports figures for the number of reported ligature incidents which are a percentage of the overall self-harm reports. Again the Oxevision ward had a reduction in the prevalence of ligatures whilst the number on the supposed control ward increased. A -48% relative percentage change is reported, again just showing that the change on the ward with Oxevision was smaller than the percentage change on the ward without Oxevision.
However, Oxehealth continually use the 44% figure in their promotional and marketing materials, incorrectly stating that this is a reduction in self-harm. For example, this screenshot from their website states that there was a 44%, 48% and 68% reduction in self-harm, ligatures and ligatures in bathrooms respectively on the Oxevision ward. None of these figures are correct and all substantially overstate what the research really showed.
Additional screenshots to places where these figures are inaccurately reported are listed at the end of this blog.[1]
It should also be noted that the reports from Coventry and Warwickshire related to this research do actually report the correct figures:
The health economics reports
We will now return to the health economics reports which were commissioned by Oxehealth. The purpose of this report is to try to quantify in financial terms how much money implementing oxevision could save. These figures are then used by Oxehealth to promote and market their technology in the UK and internationally.
Some notes
- A cost saving, no matter how high, cannot justify treatment which has a detrimental impact on patients and causes the levels of distress which have been reported by patients who have been subjected to monitoring by Oxevision.
- We will also make explicit that an effort to quantify the cost of self-harm in financial terms is inherently cold and crude. There is so much which can never be reflected within these figures. However, for brevity (and the bleak fact that healthcare across the globe often hinges on reducing people to their financial value), we will focus on the specific issues with the calculations within this report, rather than broader issues with health economics.
- There are other key issues and concerns with this analysis. For example, one of the cost savings the report considers relates to the reduced cost of a patient being on one-to-one observations when Oxevision is used instead. We assert that technology must not be used as a substitute for human interaction and relational care.
Cost savings due to the reduction in self-harm
One of the key ways the health economics reports suggest that Oxevision may save NHS trust money is due to the reduction in self-harm. However, to make these calculations they use the inaccurate 44% reduction figure. Interestingly, some of the authors of this study are the same people who conducted the study which originally identified the 44% percentage change difference (not reduction). Members of Oxehealth staff also reviewed the final manuscripts of the report, so should have been aware of what their own research showed.
https://pmc.ncbi.nlm.nih.gov/articles/PMC11389945/
Using figures for fractures
The cost saving figure for the reduction in self-harm also includes an assumption that 8% of self-harm incidents require an admission to A&E (this data wasn’t collected so is based on a different study). As well as the issues described previously, the health economics team have used the cost for admission to A&E for a minor fracture.
“First, the cost of all self-harm incidents and falls resulting in accidents and emergency (A&E) visits were, for simplification, assumed to be equivalent to minor fractures, potentially underestimating the associated costs.”
They state that this was “for simplification” however this is not sufficient justification for this decision – we argue that using the costs for minor fractures risks overestimating the cost (and therefore inflating the ‘cost-saving’ associated with Oxevision) as although self-harm methods vary and can theoretically lead to minor fractures, it would be more logical to use the costs associated with cuts or burns. These costs, as no X-Ray is required, are likely lower than the cost of a minor fracture.
The screenshot above illustrates how fracture codes were used to calculate the cost of admission to A&E for self-harm.
Conclusion
In summary, the original study did not find a statistically significant reduction in the number of reported self-harm incidents on the ward after Oxevision was implemented. There was, however, a fairly large increase in the frequency of self-harm incidents on the ‘control’ ward. Specifically, the Oxevision ward changed 44% less than the non-Oxevision ward did. That is not to be assumed to be a 44% reduction.
However, this is the figure which Oxehealth continue to present. They have even commissioned/had involvement with an economic analysis which calculates the financial value of a 44% reduction in self-harm. However, this is also based on the cost of admission to A&E for a fracture and an assumption that 8% of incidents require admission to A&E, not what was reflected by data.
This means that the cost savings they have estimated for self-harm are double what their data actually shows, and even then likely an overestimation. Despite this, Oxehealth continue to promote their product as evidence based and supported by scientific rigour.
There are some important key messages here. Firstly, all of this research has been peer reviewed and published; countless people will have reviewed publications and signed them off; and these figures have been taken at face value each time. People trust research and the processes of peer review and academic publication, but this shows how easy it is for things to slip through and mistakes to be made. As always, it should not be the job of us as patients and survivors to spot the holes in ‘evidence based medicine’.
[1] Places where the 44% reduction in self-harm is inaccurately reported:
Oxehealth publication supported by the Academic Health Science Network (now Health Innovation Networks): https://cdn.prod.website-files.com/5f567869171c90518f161723/6246c8fe4675d9f780239d1d_CWPT_Report_UK_D_1.1.pdf
Oxehealth website: https://www.oxehealth.com/
Oxehealth website: https://www.oxehealth.com/resources
An NHS AI publication: https://digital-transformation.hee.nhs.uk/binaries/content/assets/digital-transformation/dart-ed/ai-roadmap-march-2022-edit.pdf
Same data also used on the US version of the website (briefly shared then taken down):
https://www.oxehealth.com/new-home-option-2-1
Oxehealth new website: https://www.oxehealth.com/scientific-studies-2-1
An Oxford University innovation report: https://impactreport2024.innovation.ox.ac.uk/outcomes/enabling-people-to-lead-longer-healthier-lives/
An Oxehealth promotional report: https://assets.website-files.com/5f5678688f40811eed0fe944/62385b6e7e774e5e045e8a4a_IIC_Report_US_P_1.1.pdf
Another Oxehealth promotional report: https://assets-global.website-files.com/5f567869171c90518f161723/66100e47241b482782cbe961_Evidence_Report_SE_D_1.0.pdf
An Oxehealth promotional interview for their US launch: https://bhbusiness.com/2022/06/01/bhb-value-panel-a-discussion-with-oxehealth/
Another Oxford University innovation report: https://innovation.ox.ac.uk/wp-content/uploads/2024/11/OUI_Impact-Report-2024_Final.pdf
Another Oxehealth promotional document: https://assets-global.website-files.com/5f567869171c90518f161723/652f98025e7484688746547e_Workforce_flyer_SE_D_1.0.pdf
Leave a Reply