Search Menu
Photonics Media Photonics Marketplace Photonics Spectra BioPhotonics Vision Spectra Photonics Showcase Photonics ProdSpec Photonics Handbook

Duke Researchers Take Aim at Neural Network Bias

Facebook Twitter LinkedIn Email
A team of researchers at Duke University is addressing issues of transparency when it comes to the deep learning methods of neural computer vision systems. A technique the team has introduced aims to help to understand potential errors and biases in the “thinking” of deep learning algorithms. The issue, known as the “black box” problem, describes the hidden reasoning within neural networks that is largely unknown, even, in some cases, to designers. Previous attempts to shed light on the thought processes behind such decisions have considered the actions following the learning stage itself, highlighting what the computer was “looking” at rather than its reasoning.

“The problem with deep learning models is they’re so complex that we don’t actually know what they’re learning,” said Zhi Chen, a Ph.D. student in computer science professor Cynthia Rudin’s lab. “They can often leverage information that we don’t want them to. Their reasoning processes can be completely wrong.”

Instead of focusing on what the machine is looking at after the fact, the researchers’ method trains the network to show the processes of its work by demonstrating its understanding along the way, showing which concepts it’s employing to make its decision. Even with the adjustments to the network, it is able to retain the same level of accuracy as the original model, as well as the ability to show how the results are determined.

In the technique, one standard portion of a network is substituted for a new part that constrains a single neuron in the network to fire in response to standard tags and classifications that it uses to make its decision. Using a neural network trained with millions of labeled images, the researchers tested their method by feeding it images it hadn’t seen before. The researchers were able to see a readout of the network’s thought process and the unique tags through which it cycled before making a decision. 

The module can be wired into any neural network trained to decipher images, the researchers said.

The researchers connected the solution to a network designed to recognize skin cancer, which had been trained with thousands of images labeled and marked by oncologists. One of the tags the network read out surprised the researchers, they said: “irregular border.” The system was not programmed with that tag, instead developing it on its own through information it had gathered from its training images.

“Our method revealed a shortcoming in the data set,” Rudin said. “This example just illustrates why we shouldn’t put blind faith in ‘black box’ models with no clue of what goes on inside them, especially for tricky medical diagnoses.”

The research was published in Nature Machine Intelligence (
Dec 2020
artificial intelligence
The ability of a machine to perform certain complex functions normally associated with human intelligence, such as judgment, pattern recognition, understanding, learning, planning and problem solving.
convolutional neural network
A powerful and flexible machine-learning approach that can be used in machine vision to help solve difficult problems. Inspired by biological processes, multiple layers of neurons process portions of an image to arrive at a classification model. The network of neurons is trained by a set of input images and the output classification (e.g., picture A is of a dog, picture B is of a cat, etc.) and the algorithm trains the neuron connection weights to arrive close to the desired classification. At...
machine vision
Interpretation of an image of an object or scene through the use of optical noncontact sensing mechanisms for the purpose of obtaining information and/or controlling machines or processes.
Research & Technologymachine learningcomputer visionimage classificationobject-oriented image classificationobject recognitionartificial intelligenceconvolutional neural networkneural networkneural networksdeep learningmachine visionThe News Wire

Submit a Feature Article Submit a Press Release
Terms & Conditions Privacy Policy About Us Contact Us
Facebook Twitter Instagram LinkedIn YouTube RSS
©2023 Photonics Media, 100 West St., Pittsfield, MA, 01201 USA, [email protected]

Photonics Media, Laurin Publishing
x We deliver – right to your inbox. Subscribe FREE to our newsletters.
We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.