Data ethics

Social influencing

Sophisticated algorithms and advanced analytics can surface incredible insights from data.

When used positively, the benefits are evident. But such tools can also be deployed to influence or manipulate the decisions we make.

Automated decision-making

Automated decision-making also raises concerns. AI algorithms are trained how to ‘think’ using large data sets.

However, their decisions are only as good as the information they ingest, and when that data is incomplete or skewed towards a particular demographic, algorithms can amplify biases and perpetuate inequalities. This is particularly so when multiple algorithms are linked together to work alongside and learn from one another, for example in deep learning and neural networks.

Facial recognition technology

Although systems are improving fast, earlier versions were less accurate.

Research from MIT in 2018 revealed that commercial products exhibited racial and gender bias. Systems were able to identify the gender of a person from a photograph with 99 per cent accuracy if the picture was of a white man. Where the subject was a darker-skinned woman, the software was only accurate 65 per cent of the time. It is vital to understand how algorithms have been developed and their potential flaws, particularly when they can be applied to data sets for which they have not been designed.

Biased models and faulty findings

These sorts of biases against minority groups can be exacerbated where algorithms are used in techniques such as predictive analytics, whereby historic data is analysed to make predictions about the future.

Examples include:

• an insurer that tried (and failed) to monitor social media posts to gauge how dangerously a person might drive;
• credit card companies that have reportedly limited credit for individuals seeking marriage counselling due to correlation between divorce and default;
• price comparison websites that have reportedly quoted higher premiums for names implying ethnic minority status; and
• crime prediction software that increases police surveillance of marginalised groups, perpetuating bias and affecting social cohesion.

There is also an issue around transparency, for example where AI systems are used to screen candidates for jobs and reject applicants for reasons that may be unclear.

Then there are cases where algorithms just don’t work very well. A flu prediction algorithm that analysed search activity to predict where outbreaks of the virus might strike next suffered ‘model drift’, while social analytics during Hurricane Sandy wrongly placed Manhattan as the disaster hub after the worst-hit areas of Breezy Point, Coney Island and Rockaway saw limited mobile usage due to blackouts, drained batteries and limited cellular access.

‘Psychographics’, profiling and behavioural targeting

China’s social credit system, criticised by human rights groups, ranks  citizens based on social and behavioural data. It offers perks like preferential loans and quick access to doctors to those who score well, and restricted travel or access to public services to those who don’t.

Consent and opt-out

Laudable attempts to break data silos can lead to an accountability deficit. In the UK, NHS big data initiatives care.data (which was developed to aggregate health and social records) and Streams (an app to provide information to health workers in time-sensitive emergencies) were criticised for their inadequate protocols for consent, public awareness, opt-outs and accidental privacy breaches.

Secondary use

Many organisations will want to use data they have collected for another purpose, or to aggregate data before they decide what to do with it.

In the EU, the GDPR has rules about how this can be done with personal data.

• The US National Security Agency’s PRISM initiative, a surveillance programme that grew out of post-9/11 surveillance practices, acquired telephone records and ‘backdoor access’ to private electronic records on citizens held by major technology companies.
• Property records and geographic profiling were allegedly used to identify pseudonymous artist Banksy.
• Community maps used to identify properties or clarify land rights could be reused to identify opportunities for redevelopment.
• Data mining tools are identifying criminals through the ‘digital breadcrumbs’ they leave online. Digital evidence can be admissible in court, even when private.
• Mobile phone and social media data have been used to halt or throw out unfounded rape allegations; however, the victims’ commissioner for London worries women may not report cases for fear past digital communications will be misinterpreted.