15. Reliable AI to Design a Better Future - Student Presentations, Discussion & Conclusion
June 10, 2022 lectured Distinguished Students of the Course and written by Dr. Merve Ayyüce KIZRAK,
A man without ethics is a wild beast loosed upon this world. — Albert Camus
In the last lesson of the 2022 Spring semester, students were randomly divided into two groups, A and B.
The Subject of Group A: AI-empowered grading system in U.K.
The Subject of Group B: AI-empowered arrest decisions
Both topics have been covered before in the lecture. They researched these two issues as a group for 1 hour and evaluated this real-life case through the checklist below.
Checklist: Requirements of Trustworthy AI
Human agency and oversight: Including fundamental rights, human agency and human oversight
Technical robustness and safety: Including resilience to attack and security, fallback plan and general safety, accuracy, reliability and reproducibility
Privacy and data governance: Including respect for privacy, quality and integrity of data, and access to data
Transparency: Including traceability, explainability and communication
Explainability: refers to the knowledge of the internal mechanics of a model: what a “node” in the model represents and its specific relevance to the outcomes (e.g. the “age” variable determines 15% of the outcome).
Diversity, non-discrimination and fairness: Including the avoidance of unfair bias, accessibility and universal design, and stakeholder participation
Societal and environmental wellbeing: Including sustainability and environmental friendliness, social impact, society and democracy
Accountability: Including auditability, minimization and reporting of negative impact, trade-offs and redress.
Grup A: AI-empowered grading system in U.K.
Case: UK government’s A-LEVEL grading AI, developed by Ofqual, was used during the start of the pandemic.
Ethical Evaluation by Checklist
Students have the right to an education without prejudice and to shape their future. The UK government has ignored this with its grading system. To avoid such disasters in the future, authorities need to “be more inclusive and diverse in the process of creating such models and algorithms,” says Ed Finn, an associate professor at Arizona State University and the author of “What Algorithms Want.
A robust AI should not pose unreasonable safety risks, in conditions of normal use or misuse. It should be functioning appropriately and secure throughout its entire lifecycle.
To do this, it is crucial that engineers keep track of traceability, processes, and decisions made.
For most of the students that were from less advantaged schools, their scores had downgraded, while students from richer schools were more likely to have their scores raised. Professor Jo-Anne Baird, director of the Department of Education at the University of Oxford explained that:
From this, we can conclude that the system is starting to fall apart.
The right of every citizen to manage their own personal information and make decisions about it is known as privacy.
However, the data collection of this AI is suspicious. What was absent here was transparency and analysis of the algorithm’s intentions, and there was plenty of time to do so.
“There’s a lot more that can be done in terms of demonstrating proper risk and effect reduction.”
In the end, concerns about lack of transparency may diminish future faith in algorithmic systems that can benefit society. What this means is that there is no public debate or awareness of events until after they have occurred or until rumors have surfaced.
Through research, we have concluded that most of the algorithms’ explainability is close to none which caused major issues in the past. With newer algorithms, however, we see more explainability.
The way your AI professionals build out your system will determine how your algorithms interact with people of all cultures, genders, sexualities, races, and so on. Using Ai for grading is tricky because the algorithm trained by the last 3 years’ data which data they used can already be discriminant yet Aİ has a distance to every student. The case of grading particularly is a pretty sensitive topic, one mistake done by the ai due to bias can go as far as ruining a student’s future so an expert diversity team is much needed and required
The AI grading system and its bias created a massive social outcry, students were shocked to see that they received grades that were proportionally lower than what they have expected, they went on to showcase their frustration in the form of hundreds of protests, while gathered outside of the Department for Education’s building, they chanted and voiced their frustration with the system and its algorithm.
Parents of children whose papers were subjected to the system were also unhappy, they expressed how they were not notified in advance, and how the test results were kept under tight confidentiality. In some classrooms, students were not even informed that their work had been assessed by AI rather than a human.
The ordeal was mainly blamed on poor political decisions and the simple misunderstanding of data, according to Jenny Brennan, a researcher of AI and technology’s impact on society at the Ada Lovelace Institute,
“It’s more about the procedure and the issues of goals, risk mitigation, adequate scrutiny, and remedy. According to her, the algorithm in itself wasn’t necessarily wrong, but it was the wrong one to use for this type of problem, especially with its massive scale and huge effect on the population.”
Deals with fairness, liability, audibility, traceability
Every summer thousands of students take the A-levels, an exam that affects their future.
Instead of taking this exam and determining their own future, an algorithm was used to do it for them.
If worked properly, the AI system could provide a fair way of predicting the results. (exam day circumstances and the futures of the students being dependent of just a couple of hours)
However, the algorithm “predicted” the scores way lower than how students expected them to be.
The algorithm was biased.
The socioeconomic situations of the students were clearly a criterion for the algorithm. (Scores of students in less-advantaged schools were downgraded, while scores of students in richer schools were more likely to be raised.)
This kind of algorithm should be more inclusive and diverse.
Grup B: AI-empowered arrest decisions
Case: Courtrooms across the US have turned to automated tools in attempts to shuffle defendants through the legal system as efficiently and safely as possible.
Argument: AI is wrongfully sending people to jail
Example: The Killing of George Floyd
Example: How Wrongful Arrests Based on AI Derailed 3 Men’s Lives
Example: How AI Powered-Tech landed a man in jail with Scant Evidence Williams was arrested in January 2020 for allegedly stealing five watches from a Shinola store in Detroit, after he was wrongfully identified by facial recognition software. He was among the first people known to be wrongfully accused because of the software, which is an increasingly common tool for police. Michael Oliver and Nijeer Parks were wrongly arrested in 2019 after also being misidentified by facial recognition technology.
Ethical Evaluation by Checklist
Crawford and Schultz argue:
“When challenged, many state governments have disclaimed any knowledge or ability to understand, explain, or remedy problems created by AI systems that they have procured from third parties, The general position has been “we cannot be responsible for something we don’t understand.” This means that algorithmic systems are contributing to the process of government decision making without any mechanisms of accountability or liability.”
There are many private companies that market facial recognition products to state and local law enforcement.
AI systems should be secure and resilient in their operation in a way that minimizes potential harm, optimizes accuracy, and fosters confidence in their reliability
In this case, we need to hold the people or the government accountable to avoid it from happening again.
If people were held accountable, more awareness will be achieved hence, people will be more understanding of this and will be able to trust the process.
Societal and environmental wellbeing: Including sustainability and environmental friendliness, social impact, society, and democracy.
Getting wrongly convicted or sentenced can have a negative effect on society and the environment in general.
Conclusion: For these reasons, ai isn’t ready to be in the justice system just yet, much adjustment needed to be implied because freedom is not something we can put in the hands of something we don’t even understand just yet.
Recap
It characterizes AI systems and applications in the following dimensions:
People and Planet, Economic Context, Data and Input, AI Model and Task and Output.
Each dimension has its own characteristics and attributes or sub-dimensions related to evaluating policy evaluations of particular AI systems.
The phases of the AI system lifecycle can be associated with the dimensions of the OECD Framework for the Classification of AI Systems.
This mapping is useful to identify some of the key AI actors in each dimension, which has accountability and risk management implications.
Note: The actors included in the visualization are illustrative, not exhaustive, and based on previous OECD work on the AI system lifecycle.
References:
Group A
Group B
Last updated
