Smartphone photos will never be the same.
This is thanks to computational photography that will make each snap shot look like it was taken using a professional camera.
There is no denying that one of the most sought after features of any smartphone today is its camera. Ever since image sharing sites and social media platforms rose to popularity, sharing pictures of just about anything has taken over the lives of many people around the world.
Smartphone manufacturers started developing camera phones that can capture high-quality images to satisfy the needs of photo-savvy individuals. If that is not enough, some of these mobile phone giants partnered with famous camera makers to create the best camera phones of today.#MachineLearning to take over phone photography enhancement soon #Google #MITClick To Tweet
For instance, Chinese mobile phone manufacturer Huawei entered into a partnership with world-renowned camera maker Leica to produce the Leica branded, dual-camera phones P9, and the current flagship smartphone of the company which is the P10.
Other manufacturers resorted to enhancing their phones’ capabilities. One good example is the Nokia 808 PureView smartphone which is equipped with a 38-megapixel sensor that captures incredible amounts of image details.
However, despite the many mobile camera innovations we have today, people still require photo editing tools to enhance their images. To address this need, Google and MIT developed machine learning algorithms that can retouch photos in real time like professional photographers.
Machine Learning to Enhance Smartphone Pictures
According to MIT News, MIT’s Computer Science and Artificial Intelligence Laboratory and Google collaborated to make a system that automatically retouches images in the style of a professional photographer.
The machine learning system was said to produce highly-dynamic images which capture subtleties of color lost in standard digital images. Further report states that the “new system produced results that were visually indistinguishable from those of the algorithm in about one-tenth the time—again, fast enough for real-time display.”
A report from the Verge confirmed that researchers used machine learning to create their software, training neural networks on a dataset of 5,000 images created by Adobe and MIT.
Five different professional photographers retouched all images, and the algorithms used the data to analyze and learn what improvements are needed to enhance different photos. The system compared the original shots from the retouched images and mimicked the enhancements to readjust the pictures.
While the machine learning algorithms used to improve images are nothing new, the real challenge lies with slimming down the data size of the algorithms so they could be small enough to run efficiently on mobile devices.
During the early stage of the project, MIT researchers used a cell phone to send a low-resolution version of an image to a web server. “The server would then send a ‘transform recipe’ which could be used to retouch the high-resolution version of the image on the phone while reducing bandwidth consumption.”
According to Michael Gharbi, an MIT graduate student in electrical engineering and computer science and first author on both papers:
“Google heard about the work I’d done on the transform recipe. They themselves did a follow-up on that, so we met and merged the two approaches. The idea was to do everything we were doing before but, instead of having to process everything on the cloud, to learn it. And the first goal of learning it was to speed it up.”
At first, the researchers fed the system low-resolution images and increased its resolution by guessing the values of the omitted pixels. While the process produced excellent results, it would not be practical for real life usage as the low-res image leaves out too much data.
Gharbi and his colleagues used two tricks as a remedy to the problem. First was by changing the result of the machine learning system from image to a set of simple formulae for modifying the colors of image pixels. The second was by using a technique for determining how to apply those formulae to individual pixels in the high-res image.
The system was able to produce a three-dimensional grid: 16 by 16 by 8.
The 16 by 16 faces of the grip represents the pixel locations in the source image, while the eight layers stacked on top of them represents the different pixel intensities. Within each cell of the grid, the formulae which determine the modifications of the color values of the source images are contained.
Simply put, the system would not apply the filters on compressed images, rather it would impose the filtering layers on top of the shot that has not been taken yet.
The whole system now weighs around 100 megabytes, light and fast enough to operate on a mobile device. Jon Barron, a Google AI scientist, was quoted as saying:
“This technology has the potential to be very useful for real-time image enhancement on mobile platforms. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”