Last Updated on July 16, 2022 by pg@petergamma.org
Since years, consumer grade sports sensors attract scientists from around the world, like the group of the Cardiologist Milind Desai, MD from Cleveland Clinic in Ohio:
a group of the Institute for Intelligent Systems Research and Innovation, Deakin University Melbourne:
the Swiss Federal Institute of Sport in Magglingen:
Rob ter Horst, PhD. from “The Quantified Scientist”:
and the MEDITATION RESEARCH INSTITUTE SWITZERLAND:
Can these consumer grade devices not also be used for clinical & research applications? When the Polar OH1 came out, DC Rainmaker and 5krunner entiteled the Polar OH1 as a very accurate, if not the most accurate optical heart rate sensor they tested.
2019, a paper from the Institute for Intelligent Systems Research and Innovation, Deakin University Melbourne came out which investigated the accuracy of the Polar OH1:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0217288
They found out that the Polar OH1 can be used instead of an ECG or chest strap with an accuracy value of 95% to 99%. But is this accurate enough for our applications?
After the Polar OH1, it s successor Verity Sense came out. It is hard to believe, that a successor of a sports sensor is less accurate than his predecessor. Therefore the validation paper for the Polar OH1 should in theory also apply to the Verity Sense. Then, Polar increased the running time of the Verity Sense by only firmare update:
How was this firmware update performed? Does it reduce the accuracy of the Verity Sense? DC Rainmaker stated that this was achieved by cleaning up algorithms and other software optimizations, and that he and countless reviewers did see zero accuracy changes before and after the firmware update of the Verity Sense.
A year ago, Peter H. Charlton, PhD. launched the the ‘Respiratory rate algorithms for wearables’ project:
It uses motion artefact removal algorithms to improve the signals of optical heart rate sensors. Also HeartPy:
uses such algorithms. Did Polar use such algorithms to clean up the Verity Sense signal to enhance the battery life of the Verity Sense? If so, Polar could also have risen the accuracy of the Verity Sense instead of prolonging the battery life. Was the longer battery life achieved by an internal reduction of the sampling rate?
DC Rainmakers answered this question on his blog: «I’m tired of having to refute your claims». DC Rainmaker does not have to refute your claims, but he cannot convince us either by this statement. Polar looses customers which depend on an optical heart rate sensors with high quality validated numerical an statistical data about it s accuracy trough this firmware update:
The group of the Institute for Intelligent Systems Research and Innovation, Deakin University Melbourne validated the Polar OH1 with a 64 channel nautilus device which costs at least 50 000 USD:
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0217288
Isn’t it a pity to spoil an excellent validation paper, which alleady exists for the Polar OH1 with an intransparent firmware update for the Verity Sense? A new validation paper becomes necessary for the Verity Sense. We keep using the Polar OH1, until new validation data are available. Polar, keep selling the Polar OH1. It is validated for it s accuracy in an excellent PLOS ONE paper, but not the Verity Sense.
Rob ter Horts PhD. published a ranking of more than 50 sports sensors on his youtube channel:
Polar OH1 and Verity Sense do not belong to the most accurate devices anymore according to Rob. In his ranking, the most accurate devices are Apple watches an Huwei watches. Unfortunately, Rob does not publish any numerical and statistical data outside of his Youtube channel. Can Rob not publish his data? It does not have to be a scientific paper, a Wiki site would be sufficient. Others might follow Rob, until sufficient numerical and satistical data are available, to get statistical significant values. Trough such a project, sports sensor accuracy validation would become a community project. Hopefully also DC Rainmaker and 5KRunner would join this project. A standard protocol would be necessary to test sports sensors.
Another issue is, that Rob ter Horst uses a controversial paper of the Swiss Federal Institute of Sport in Magglingen as a scientific legitimacy to use the Polar H10 as a reference device for his studies:
The Magglingen paper says that the accuracy of the Polar H10 is 99.6 % for the test they performed in the paper. The paper made Rob ter Horsts heart skip a beat, and he immediately published a Youtube video about this topic:
Rob ter Horst entiteled his Youtube video: «Polar H10, 99.6 accurate, best heart rate». But this interpretation of the paper of the Swiss Federal Institute of Sport Magglingen is controversial. Since years we are promoting the idea to use 16 bit 3 lead high quality ECG devices as reference devices to validate consumer grade heart rate sensors. The Swiss Federal Institute of Sport Magglingen made us brainstorm for a week validation methods for sports sensors to save what can be saved, after Rob ter Horst released a promo video about the Polar H10 with an accuracy of 99.6%, which seemingly seem to make ECG validation devices unneccessary, since they seem not to be helpful according to the Magglingen paper.
If the Polar H10 with a seemingly accuracy of 99.6 % is “the best” as Rob ter Horst claims, also for scientific purposes, which was also claimed by Rob ter Horst on his Youtube channel, also doctors in the hospital would choose a Polar H10 chest strap for theirs patients, and scientist the sports strap sensor in their laboratory. The Swiss Federal Institute of Sport Magglingen paper states an accuracy of 99.6 % to the Polar H10. But this accuracy value is only correct in the context of the paper.
A paper of the Cardiologist Milind Desai, MD found out that the accuracy of the Polar H7 is 98 % when compared to a Cardiac Science ECG device:
The paper of Milind Desai uses a different protocol than the paper from Magglingen, therefore the data cannot be compared. It is hard to believe that the accuracy of the Polar H7 has risen from 98 % to 99.6 % to the Polar H10. An increase in accuracy of 1.6 % from the Polar H7 to the Polar H10, we cannot believe in this huge increase in accuracy.
The high accuracy value of the Polar H10 might be caused by the protocol of the Magglingen paper. We suppose, that the accuracy of the Polar H10 is also in the range of 98 %, when compared to an ECG device, when motion artefact do not play a role, when for instance Milind Desais protocol would have been chosen in a study.
Under this assumption, Rob ther Horst, DC Rainmaker and 5KRunners reference device Polar H10 would have an accuracy of 98 % when compared to a Cardiac Science ECG device. Rob ter Horst used high quality research grade sleep EEG caps for sleep studies. Why does he not use the most high quality reference devices like g.Nautilus or Adinstruments Equivital for his validation studies, as well as DC Rainmaker and 5KRunner? This would massively increase their credibility.
Rob ter Horst is a talented scientist with a PhD. in Computer Science. Why does he not use a stackable 8 channel 16 bit Rasperri PI hat with HeartPI Python software and motion artefact removal as reference device:
and sells it to DC Rainmaker, 5KRunner and us. The costs for this device are only a few hundert dollars, and the device has the potential to find friends around the world, since it offers for a few hundert dollars an accuracy which can compete with Biopac devices which costs 4000 USD. Does such a project not make much more sense for Rob ter Horst than measuring the accuracy of each Aura ring and publish it on Youtube?
According to Rob ter Horsts plot, Apple watches and Huawei watches are the most accurate devices. But these devices to not allow raw sensor data access as far as we know, which allows calculations to reach a maximal accuracy values.
The this is ANT+ protocol does not allow sensor raw data transmission, which is written somewhere in the this is ANT+ forum. Is it not better to choose a device, which offers this option by the manufacturer, like Polar, for instance in the Polar SDK or in the Polar Sensor Logger App?
What a huge incease in quality is reached by processing sensor raw data is shown in this demo video from the Iworx ECG toolbox with Rat ECG:
After years of dealing with Garmin watches and Polar sensors, our hearts make a jump when looking at the ECG toolbox from Iworx. Every ECG peak can be analyzed in detail. Is this not much more fun than dealing with intransparent .fit files and how to make those readable for a data analysis software?
Waiting for more accurate optical heart rate monitors has become somehow frustrating. According to Rob ter Horsts plot, the most accurate devices from Apple or Huawei reach an accuracy which is near to his Polar H10 reference device.
According to Milind Desais paper:
this still has only a accuracy of 98 %, assuming that the accuracy of the Polar H7 is near to the accuracy of the Polar H10. The Body guard first beat has a validation paper:
but the Firstbeat Body Guard is not as accurate as a 16 bit 3 lead research grade ECG device. BG2 detected on average 99.95% of the heartbeats (0.05% of beats missed), whereas the Polar H10 is 99.6 % correct. We have a new ranking here with two papers. Do these two paper measure the same parameters? If so, we have a new winner.
1. best heart rate monitor: Firstbeart Body Guard 2 99.95% accuracy
2. second best heart rate monitor: Polar H10 99.6 % accuracy
So Rob ter Horst, PhD. statement that the Polar H10 has the «best heart rate» fails also here, the Polar H10 is also in this ranking not the best.
The Polar H10 also has only two ECG electrodes. Whe suppose that this architecture sets a limit to the accuracy of these devices, and that they will not reach the accuracy of 16 bit 3 lead research grade ECG devices.
ECG devices usually have 3 ore more channels.
Why not to increase the accuracy of the Polar OH1 by using three or more Polar OH1 sensors, and integrate those for instance into a chest strap? Or to stick three OH1 or more sensors to your chest according to an ECG placement layout?
According to Polar, the left or right forearm or upper arm are the locations they have verified that work best. Alternatively, we suggest to use these posititions for a multi-sensor setup which can compete with clinical or research grade ECG devices.
There is a Polar SDK available. We are convinced that it is possible to build such a multi Polar OH1 device, which has the accuracy of a 16 bit 3 lead research grade ECG device, which can be used to for clinical & research applications.
It is not urgently necessary to build a special optical heart rate sensor device with consists of three Polar OH1 sensors for this purpose. It is sufficent to combine sensor data from three devices. But it would be highly desirable to know how accurate such a device is which joins sensor data from three OH1 devices, and to have a high-quality validation paper about it.
We are physiologists and not interested in software and hardware development. Therefore we are looking for a diploma thesis engineer student who is interested in this promising project to develop such a device. A lot of information to realize it can be found on this blog, and we can also be reached by e-mail: pg@petergamma.org.