Image Colorization System Uses AI to Direct Its Choices

A data-driven image colorization technique uses AI and deep learning to colorize images in real-time, providing even novices with the ability to quickly colorize gray scale images with satisfactory results. The system is guided by a Convolutional Neural Network (CNN), which propagates user edits by fusing low-level cues with high-level semantic information learned from large-scale data. The colorization is performed in a single feed-forward pass, enabling real-time use.

The CNN is trained to directly map gray scale images, along with user inputs, to the output colorization. The user provides guidance by adding colored points, or “hints,” which the system then propagates to the image.

The CNN also learns common colors for different objects and makes appropriate recommendations to the user. To guide the user toward efficient input selection, the system recommends plausible colors based on the input image along with user inputs.

Although the CNN is trained with ground truth natural images, it is able to colorize objects with different, or even unlikely, colorizations. For example, although trained to color elephants as brown or gray, a pink elephant is not off limits for the CNN, if that’s what the user wants.

The proposed system uses AI to colorize a gray scale image (left), guided by user color 'hints' (second), providing the capability for quickly generating multiple plausible colorizations (middle to right). Photograph of Migrant Mother by Dorothea Lange, 1936. Courtesy of Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C.

This system improves upon previous automatic colorization systems by enabling the user, in real-time, to correct and customize the colorization. Existing automatic methods of colorization aim to choose a single color and do not allow a user to specify his or her preference for an alternate color choice.

To evaluate the system, the researchers tested their interface on novice users, challenging them to produce a realistic colorization of a randomly selected gray scale image. Even with minimal training and limited time — just one minute per image — the users quickly learned how to produce colorizations that often fooled real human judges in a real vs. fake test scenario.

The research was conducted at University of California, Berkeley. The code for the research has been made available by the research team.

The research will be presented at SIGGRAPH 2017, July 30-August 3, Los Angeles Convention Center in Los Angeles.

Real-time user-guided image colorization with learned deep priors. Courtesy of University of California, Berkeley.