I assume most of the readers of this blog are video / photo / gadget / phone / camera geeks. I am sure you didn’t miss the reviews of the latest Apple iPhone 7 Plus and Google Pixel phones. They have a lot in common, but there is one major aspect that is interesting for our applications: things are slowly moving from photography to computational photography. We are no longer just capturing light coming from optics and applying some minor processing to the pixel values to make the picture more pleasant to the viewer.
Phones must be slim and light and yet we still expect to have near DLSR quality. So, now computational photography comes into play. The iPhone 7 Plus, for example, uses two different cameras to calculate a depth of field and then tries to simulate the “bokeh” effect via software you would normally get in bulky professional cameras, by using fast optics at a wide aperture.
On the other side, when you hit the button on the Pixel phone, it is capturing a bunch of pictures and then decides what to keep from every picture in order to give the user the final result.
This challenges the concepts of originality and authenticity. The light captured by the camera is no longer the output of the photography process, but just the first step of a more complex process based on a multitude of factors. There is little doubt that this is just the beginning of a trend which will explode in the next few years.
Let’s talk about the ancestors of these techniques: the panoramic mode and the HDR.
The panoramic mode tries to compensate the shallow field of view of camera optics by stitching multiple pictures covering a scene up to 360 degrees. But it’s not as easy as it looks: there is some processing, equalizing, merging, and cropping to make them fit together. Many implementations of the panoramic mode suffered bad stitching artifacts or other problems, but they evolved quickly and panoramas taken by modern devices have very good quality. Can we consider a panoramic image taken by a smartphone as an original? Does it represent a truthful representation of reality? It depends on what we are looking for and the purpose of the pictures. If the stitching algorithm didn’t work properly and the horizon has steps, and the sea level is going up and down, then we cannot consider it a reliable representation of reality.
It must be added, though, that digital photography has always been more or less “computational”. If we really want to be strict, because of the CFA filter and CFA interpolation, we can say that the vast majority of the pixels in a digital image are not “original” but calculated according to the nearby ones.
The HDR mode tries to compensate another technical limitation of cameras, which is the limited dynamic range. With current technology, it is difficult to properly render images which have both very bright and very dark areas. Our eyes are much more flexible in this respect. HDR mode then takes multiple pictures with different exposures and merges different parts in order to avoid saturation. While in many cases it works well, if the subject is moving, there can be strange artifacts, especially ghosting artifacts. Still, there are often strange artifacts and strange effects. Also here we may have some doubt about the authenticity of the pictures. I remember HDR pictures with ghosts and two-legged dogs. And yet in many modern devices, HDR is left on or in auto-mode by default.
The panoramic mode and the HDR pose many different challenges for the authentication of images. Depending on the processing, different forensic operators can give unusual results. Some basic analysis is not easily feasible: for example, it is not possible to check if a picture matches the standard resolution in a panorama since every image will have a different resolution depending on the swiped area. The panorama is composed of stitched-together images acquired by slightly different parts of the sensor, making the analysis of the sensor pattern noise or PRNU difficult, if not impossible.
I think that the computational photography techniques used by the latest devices provide significant challenges for the authentication of images. On the other side, they probably leave some specific artifact that research in the field of image forensics sooner or later will be able to exploit.
