In our previous post, Why Accuracy Matters: VisionCheck in the Real World, we shared the performance of VisionCheck when used by both everyday users and expert users. Today, we are taking this a step further by testing VisionCheck on two new ethnic groups as part of our ongoing efforts to ensure that our technology works equally well for all.

Why does ethnicity matter?

Human eyes share the same fundamental structure worldwide. However, ethnic variations exist, such as iris colour and pigmentation, which can affect optical measurements of physical phenomena, including eyesight assessments. Therefore, systems must be developed and tested across the entire target population. Key among these is ensuring that the data used is relevant, sufficiently representative, and, where possible, free of errors, and that it has appropriate levels of accuracy and robustness. This principle aligns with our broader commitment to inclusive AI development, as discussed in our "Why OptikosPirme Embraces the AI Act" blog post.

Why This Study?

VisionCheck is our product for fast, picture-based estimation of myopia and hyperopia. While it has been trained on a diverse set of users, most of our historical training data comes from individuals with lighter skin tones. This means the model has limited exposure to certain eye appearance variations that can occur with darker skin colours.

Through our partnership with Sightsavers, we consistently gather data from countries like Pakistan and Nigeria. The data is collected by opticians, whom we regard as expert users, meaning they are trained in the proper use of our products, producing specific data. While these images were predominantly gathered for our Argus product, which provides precise eyeglass prescriptions from any location, they also offered a valuable opportunity to evaluate VisionCheck’s performance on populations it has not been explicitly trained on.

We specifically wanted to assess how VisionCheck performs with natural variations in eye characteristics across different populations, something we have observed can vary noticeably.

Study Details

We evaluated VisionCheck on a subset of expert-user data collected from Pakistan and Nigeria. For this preliminary test, we assessed a subset of the data collected: a total of 224 randomly selected eyes, of which 143 were from Pakistan and 32 from Nigeria. These represent only part of the data gathered from each region.

Overall, VisionCheck performs on par for eyes from Pakistan and Nigeria compared to those from European populations. There is a slight drop, in particular for eyes from Nigeria. This aligns closely with expectations.

Key Observations

Strong Generalisation Without Training Data
Despite having never been trained on data from these ethnic groups, VisionCheck still performed at a high level — especially on the Pakistani dataset, where both accuracy and F1 matched closely with results from our original validation set.

Slightly Lower Performance on Nigerian Dataset
While the results for Nigeria were still good, they were lower compared to those in Pakistan. This aligns with our expectation that more pigment affects our measurements, reducing the signal available for the model to infer results.

Expert Users Help Accuracy
All data came from trained expert users, which likely reduced noise from poor image capture and helped VisionCheck perform optimally.

Next Steps

These results are encouraging, as they demonstrate that VisionCheck can operate reliably even outside its training distribution. However, there’s room for improvement.

Additionally, our collaboration with Sightsavers has not only validated VisionCheck’s robustness but also highlighted the importance of continuous model expansion to serve diverse populations and avoid bias in our system. By including these new datasets in training, we aim to make VisionCheck even more reliable for everyone, everywhere.

As part of our upcoming scheduled model retraining, we will integrate these new datasets into our training pipeline. This will help VisionCheck handle a broader range of eyes even more effectively.

We invite organisations and individuals who share our passion for advancing vision care to join our research efforts. Your expertise and insights can help us refine our technology and make an even greater impact on global eye health.

Reach out to learn more about R&D collaborations.

‍