Images Alone Form a Dynamic 3D City Model Using Machine Learning

“VarCity,” a technology platform that can create 3D city models using image data alone, can evaluate and automatically combine images from all types of sources, including aerial photography, panoramic images taken with special vehicles, and photos published on social networks and internet platforms. It can also use video material, such as from public webcams.

Based on machine learning algorithms, VarCity can recognize image content such as buildings, windows, doors, streets, bodies of water, people and cars. It “knows,” for example, which streets are one-way only.

Using machine learning algorithms, the model can differentiate between buildings, streets, vegetation and bodies of water. Courtesy of ETH Zurich.

In a triangulation approach, researchers used several pictures of the same object from various angles to create a 3D model from the image data. They analyzed the distances and angles in the various images to precisely locate the corner points of the object in the relevant space.

To showcase their work, the researchers created a 3D model of the city of Zurich.

The VarCity technology, developed at ETH Zurich’s Computer Vision Lab, is capable of evaluating huge volumes of data.

“The more images and videos the platform can evaluate, the more precise the model becomes,” said researcher Kenneth Vanhoey. “The aim of our project was to develop the algorithms for such 3D city models, assuming that the volume of available images and videos will also increase dramatically in the years ahead.”

The technology has many possible applications, including in urban and transport planning. Counting traffic could be done exclusively with cameras and would no longer require sensors to be installed on the road.

The model also recognizes facades and windows (highlighted in yellow and red in the buildings to the front left of the picture). Courtesy of ETH Zurich.

“The whole thing goes so far that we can perform a highly detailed analysis of traffic flows on the street and also measure pedestrian traffic, even in real time depending on the application,” said Hayko Riemenschneider, VarCity project manager.

“Our technology is dynamic. It brings life, people and vehicles to 3D city models,” he added.

The technology was devised in accordance with the “privacy by design” principle, which ensures that the city models are always anonymized. People and vehicles can be shown as avatars, making VarCity compatible with data privacy.

Several startups have been founded to market potential applications for the technology. Spectando offers virtual building inspections for the real estate market; Casalva performs virtual damage analyses on buildings for insurance companies; Parquery, a parking management system, works exclusively with cameras installed near parking spaces to evaluate video data in real time and determine which parking spaces are free and whether a car has been mistakenly parked across two spaces. UniqFEED identifies perimeter advertising and individual players during television broadcasts of sports events, allowing it to calculate how long they are visible during the broadcast and to personalize the perimeter ads for various user groups.

This is VarCity. Courtesy of ETH Zurich.