Duke Researchers Take Aim at Neural Network Bias

Facebook X LinkedIn Email
A team of researchers at Duke University is addressing issues of transparency when it comes to the deep learning methods of neural computer vision systems. A technique the team has introduced aims to help to understand potential errors and biases in the “thinking” of deep learning algorithms. The issue, known as the “black box” problem, describes the hidden reasoning within neural networks that is largely unknown, even, in some cases, to designers. Previous attempts to shed light on the thought processes behind such decisions have considered the actions following the learning stage itself, highlighting what the computer was “looking” at rather than its reasoning.

“The problem with deep learning models is they’re so complex that we don’t actually know what they’re learning,” said Zhi Chen, a Ph.D. student in computer science professor Cynthia Rudin’s lab. “They can often leverage information that we don’t want them to. Their reasoning processes can be completely wrong.”

Instead of focusing on what the machine is looking at after the fact, the researchers’ method trains the network to show the processes of its work by demonstrating its understanding along the way, showing which concepts it’s employing to make its decision. Even with the adjustments to the network, it is able to retain the same level of accuracy as the original model, as well as the ability to show how the results are determined.

In the technique, one standard portion of a network is substituted for a new part that constrains a single neuron in the network to fire in response to standard tags and classifications that it uses to make its decision. Using a neural network trained with millions of labeled images, the researchers tested their method by feeding it images it hadn’t seen before. The researchers were able to see a readout of the network’s thought process and the unique tags through which it cycled before making a decision. 

The module can be wired into any neural network trained to decipher images, the researchers said.

The researchers connected the solution to a network designed to recognize skin cancer, which had been trained with thousands of images labeled and marked by oncologists. One of the tags the network read out surprised the researchers, they said: “irregular border.” The system was not programmed with that tag, instead developing it on its own through information it had gathered from its training images.

“Our method revealed a shortcoming in the data set,” Rudin said. “This example just illustrates why we shouldn’t put blind faith in ‘black box’ models with no clue of what goes on inside them, especially for tricky medical diagnoses.”

The research was published in Nature Machine Intelligence (

Published: December 2020
machine learning
Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on the development of algorithms and statistical models that enable computers to improve their performance on a specific task through experience or training. Instead of being explicitly programmed to perform a task, a machine learning system learns from data and examples. The primary goal of machine learning is to develop models that can generalize patterns from data and make predictions or decisions without being...
computer vision
Computer vision enables computers to interpret and make decisions based on visual data, such as images and videos. It involves the development of algorithms, techniques, and systems that enable machines to gain an understanding of the visual world, similar to how humans perceive and interpret visual information. Key aspects and tasks within computer vision include: Image recognition: Identifying and categorizing objects, scenes, or patterns within images. This involves training...
artificial intelligence
The ability of a machine to perform certain complex functions normally associated with human intelligence, such as judgment, pattern recognition, understanding, learning, planning, and problem solving.
convolutional neural network
A powerful and flexible machine-learning approach that can be used in machine vision to help solve difficult problems. Inspired by biological processes, multiple layers of neurons process portions of an image to arrive at a classification model. The network of neurons is trained by a set of input images and the output classification (e.g., picture A is of a dog, picture B is of a cat, etc.) and the algorithm trains the neuron connection weights to arrive close to the desired classification. At...
neural network
A computing paradigm that attempts to process information in a manner similar to that of the brain; it differs from artificial intelligence in that it relies not on pre-programming but on the acquisition and evolution of interconnections between nodes. These computational models have shown extensive usage in applications that involve pattern recognition as well as machine learning as the interconnections between nodes continue to compute updated values from previous inputs.
deep learning
Deep learning is a subset of machine learning that involves the use of artificial neural networks to model and solve complex problems. The term "deep" in deep learning refers to the use of deep neural networks, which are neural networks with multiple layers (deep architectures). These networks, often called deep neural networks or deep neural architectures, have the ability to automatically learn hierarchical representations of data. Key concepts and components of deep learning include: ...
machine vision
Machine vision, also known as computer vision or computer sight, refers to the technology that enables machines, typically computers, to interpret and understand visual information from the world, much like the human visual system. It involves the development and application of algorithms and systems that allow machines to acquire, process, analyze, and make decisions based on visual data. Key aspects of machine vision include: Image acquisition: Machine vision systems use various...
Research & Technologymachine learningcomputer visionimage classificationobject-oriented image classificationobject recognitionartificial intelligenceconvolutional neural networkneural networkneural networksdeep learningmachine visionThe News Wire

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.