Artificial Intelligence (AI) is already altering the way we live and work, providing both opportunities to grow and challenges to overcome. Yet, Australia is hesitant in its attitudes towards AI. A recent study surveying how individuals in 31 different countries feel about AI revealed that Australia was most worried, with 69% of respondents (compared to the global average of 52%) saying that they felt nervous about the technology,[1] and only 44% of Australians indicating that they trust AI systems not to discriminate or show bias.[2] These results provide a useful indication of the public consciousness as to one of the biggest risks and dangers associated with the use of AI: the occurrence of algorithmic bias.[3]
We recently reported on how the circumstances surrounding the training of AI systems can lead to the occurrence and perpetuation of bias in the operation of AI algorithms (using the examples of sex and racial bias), which may become evident in an AI system’s outputs or outcomes, and how this presents issues in the application of AI in the life sciences and healthcare sectors. However, this is not limited to these sectors, with issues pertaining to bias also relevant to the application of certain AI systems in the criminal justice system,[4] education,[5] recruitment,[6] and child protection,[7] just to name a few.
The existence of algorithmic bias is an issue not only for those making use of AI (such as developers and user organisations), but also for those who are impacted by the outcomes of AI-driven decision making. In this article, we take a deep dive into how algorithmic bias can arise, as well as the practical steps that organisations can take to resolve algorithmic bias, including where AI systems have been licensed from third party providers.
How Does Algorithmic Bias Arise?
Ideally, the goal of any AI system should be to produce useful output that is as accurate, fair, ethical and equitable as possible; but sometimes, AI systems can produce results that fall short of these expectations, producing results that may be inaccurate and/or unfair. Where this occurs, it is generally suggestive of some sort of bias within the relevant algorithm, which bias can be caused by a variety of factors, including data quality and issues with data collection, idiosyncrasies in the underlying algorithm and its operational parameters, or even human involvement in the process that is enhanced by AI (for example, selection bias in the procurement of training data sets, the selection of which algorithms are used to operationalise the relevant AI system, or issues with the interpretation of AI outputs).
The most common cause of bias in AI systems, however, is where the data used to train an AI system are in some way inadequate. Generally, data used to train an AI system are procured from a raft of different sources, including, for example, being scaped indiscriminately from the internet. It is then fed into an AI system and a learning algorithm is implemented in order to teach the system to generate (with varying degrees of user guidance) the desired output, based on statistical connections and inferences that the system draws from its training data and the relevant user input.
However, the fatal flaw in this process is that, because an AI system and the quality of its output is generally only as good as the data it was trained on, an insufficient quantity or breadth of data used to train an algorithm will likely compromise that quality. Further, if there is a lack of ‘diversity’ in the data (where the data do not incorporate diverse demographics sufficient for the AI system to make accurate or appropriate predictions in accordance with its overriding purpose), this could lead to an AI system generating output that is biased. If the training data is reflective of the social and cultural biases of human society generally, then this will likely result in those biases being reproduced in the form of the relevant AI system’s output.
Practically, the risk that this presents is not only legal in nature (possibly exposing organisations to liability where, for example, an AI system is deployed to make decisions that ultimately discriminate against or affect the rights of certain (classes of) persons), but is also reputational and existential. The unaddressed perpetuation of bias in an AI system can lead to dramatic outcomes, like a decrease in access to products and services and poorer outcomes for certain demographics, all of which will likely have an effect on the willingness of people to engage with AI systems and the organisations that deploy or use them. For example, an AI system designed to predict heart disease risk that was trained on majority male data (acknowledging that the substantial bulk of medical research to date is based on male anatomy and presentation),[8] could fail to correctly predict heart disease risk in women (either by overestimating or underestimating risk), potentially leading to poorer health outcomes for women as a result of being ‘overlooked’ by the AI system as a result of its bias.
How Can Organisations Resolve Algorithmic Bias?
The importance of identifying and resolving algorithmic bias is clear. The Australian Federal Government’s recent discussion paper on the safe and responsible use of AI comments that those designing or implementing AI systems need to design, test and validate the systems to correct for bias and potential harms.[9]
However, before any mitigation strategies can be adopted, it is crucial to understand the relevant AI model and ensure that it is, to the greatest extent possible, explainable, meaning that human users can comprehend how the system works and why the system produces certain results or makes certain decisions, as well as the impact of potential biases on those results.[10] This may be easier said than done in the case of complex AI systems, such as neural networks (AI systems that process data using interconnected groups of ‘nodes’ analogous to neurons in the human brain), where the internal workings and decision-making processes may not be easily understandable or may even be completely opaque. Nonetheless, as explainable AI facilitates trust and confidence, it is important for organisations to understand any AI product they use or develop, especially how it learns, draws inferences and makes predictions based on its training data. It is through this understanding that a user will also be able to identify any algorithmic bias and then seek to mitigate or resolve it.
In light of the fact that every AI system is trained on its own unique data set, it is likely that appropriate strategies to effectively mitigate bias will differ depending on the system in question, its size, complexity, capability and explainability, and a ‘one size fits all’ approach may not deliver the best outcome. That being said, although it may not be possible to completely eliminate algorithmic bias, where an organisation is concerned about algorithmic bias and its consequences, or indeed, identifies bias in its AI system’s output, the following practical steps may be taken to investigate, mitigate and (hopefully) resolve that bias.
Define Fairness Criteria
Organisations should start by defining what ‘fairness’ (generally defined to refer to impartial and just treatment) means in the context of the overriding purpose of the relevant AI system and the output it produces. Due to the potential for AI systems, particularly when used in a decision-making context, to affect lives, it is essential to have a management framework that quantitatively assesses the fairness of that AI system and how it operates.
As the notion of fairness is quite subjective (differing depending on legal system, culture, etc.) and broadly philosophical, fairness could be defined in various ways, including by reference to the legal precepts of discrimination, or the preferred outputs/outcomes of the AI system itself as it relates to certain individuals or groups, based on certain attributes (for example, legally protected attributes such as sex, race and disability status). Ideally, an algorithm should behave in such a way that it treats every person equitably, and so observing whether or not an algorithm treats people differently based on those attributes is a helpful way in which to identify (and then seek to mitigate) unfairness. However, the way fairness is defined in the relevant context will largely depend on the relevant use case and what the AI system is designed to achieve. For example, ‘equality’ is not likely to be an appropriate fairness metric for an AI system that needs to understand and cater for the differences between the sexes in order to achieve equitable outcomes (e.g. where differences between the sexes are relevant to disease presentation and thus relevant to achieving improved health outcomes for patients regardless of sex).
In addition, an important element of ensuring fairness is to have meaningful diversity in teams developing, and assessing the fairness of, AI systems. A team in charge of AI development, bias mitigation and monitoring fairness will have to be carefully curated, as a culture where minority voices are heard and encouraged will facilitate improved identification of potential fairness-related issues that may simply not have been considered in a team devoid of diversity. This is also important given that what a non-diverse group may view as fair is likely to differ from what a diverse group, with multiple different perspectives, may consider as fair, such that a diverse group may better and more accurately define what fairness means in the context of the relevant AI system.
Data Review
Next, organisations should review all of the data with which the AI model has been, or is to be, trained (though this may be easier said than done). It is pivotal that the data are representative and diverse, covering all relevant groups and demographics. If there is any obvious under-representation of a demographic identified in the data, the collection of additional data to address this gap should be considered. Depending on the size and functionality of the AI tool, this may be a lengthy and difficult process; however, it is a required one in order to avoid an algorithm that may perpetuate bias.
Where it is not possible to collect additional data, the limitations of the training data should be made clear to all who use the system, and ongoing work should be done to try and rectify the deficit. For instance, if the training data for an AI system used in life sciences or healthcare did not include data related to pregnant women (and a study on pregnant women could not be ethically justified) then, to the extent the AI system is used in a real world context for pregnant women, evidence should be gathered from those experiences to update the training data.
Attribute Selection
Organisations should then identify any relevant attributes that could be correlated with bias (for example, sex or race) in the relevant data set, as it is essential to be aware of the potential impact of these attributes on the relevant AI system’s outputs. If there is an acknowledgement that a data set is limited in terms of its diversity and attribute representation, then that organisation needs to be conscious that the outputs/outcomes of the AI system may be biased, and it behoves the relevant organisation to take the steps set out below in attempting to resolve that potential bias.
Measure Bias
Once the AI system has been deployed, organisations should use the fairness metric developed previously to assess bias in the relevant system. For example, biased outcomes may be able to be identified by the volume of false positive results given in relation to a particular demographic, or the manner in which the AI system ostensibly disregards certain individuals or classes of person based on certain attributes. In order to place a control into this assessment, organisations should assess performance across different demographics and attribute combinations.
Bias Mitigation And Ongoing Monitoring
If algorithmic bias is detected in an AI system, then the organisation should take measures to mitigate that bias by implementing strategies that may include:
After implementing bias mitigation techniques, testing and validation of the relevant system’s performance are imperative to check that these techniques have in fact reduced bias. It is also important to continue to monitor the system’s performance, and feedback from users, to identify areas for improvement. Finally, regular audits and reviews should be performed to identify any new instances of bias or changes of behaviour in the system itself, which processes should be carefully documented.
While the complete elimination of algorithmic bias may not always be achievable, continuous efforts to address this issue will lead to more ethical and equitable applications of AI over time.
Outside The AI System
In addition to the above steps, which focus on the AI system itself, organisations should also ensure that the parts of the organisation that surround the relevant AI system are able to assist in reducing the presence of algorithmic bias and the likelihood of discriminatory outcomes. For instance, organisations should ensure that they assign specific individuals to monitor the relevant AI system and its outputs, in order that biased and inaccurate outcomes can be detected and mitigated proactively. In addition, those parts of an organisation that interact with the relevant AI system should be trained in the concepts of bias and discrimination in order to be able to identify such issues in the system’s outputs. Having the right personnel involved in AI development and deployment is an important aspect of minimising bias.
Further, AI systems that are customer-facing also raise questions around bias and discrimination, not necessarily in respect of the AI system itself, but in terms of the ease with which it may be accessed by a diverse number of users.[11] Making AI systems accessible, non-discriminatory and user-friendly is a key imperative for any organisation that implements AI in its customer-facing operations, and speaks directly to the global focus toward ensuring diversity and inclusion in industry.
Third party Purchased / Licensed AI
The issues pertaining to bias in AI systems apply to any sort of system that learns from training data, whether that system was developed by an organisation, or purchased or licensed from a third party (Provider). Even where an organisation is purchasing or licensing an AI system from a Provider, that organisation will remain ultimately responsible for it, and will also be accountable in most instances for the effect that an AI system has on the lives of the people that interact with it. For example, in the life sciences and healthcare sectors, an AI-driven diagnostics tool could misdiagnose a patient, causing them to suffer loss. In that situation, the aggrieved individual is likely to seek a remedy from the organisation that misdiagnosed them using the AI tool (e.g. a hospital), rather than the Provider.
Acknowledging that, in this context, the user organisation will generally lack the technical permissions or capabilities to assess and mitigate bias in the Provider’s system, those user organisations should ensure that their contractual arrangements with Providers assist, as far as is reasonable, in the mitigation of issues pertaining to bias, including by:
Looking Forward
Despite a majority of Australians remaining hesitant towards AI and lacking trust in its ability to remain unbiased, over 60% of Australians surveyed agreed that products and services using AI will profoundly change their daily life within the next three to five years.[13] As a result, strategies to mitigate algorithmic bias should be front-of-mind for developers, creators, providers and user organisations. By implementing the above practical steps, involving both the AI system itself as well as the people surrounding it, it is possible to reduce the occurrence and consequences of algorithmic bias.
