Apollo is the first application to use the depth data of portrait mode photographs, to realistically add light sources to a scene. Development of the app began as an experiment back in November 2017, when we first got our hands on a brand new iPhone 8+. We wanted to see what could be achieved by taking advantage of the depth information of portrait photos. Our hypothesis looked simple: if depth information can be superimposed on a 2D photo, it should be possible to re-illuminate objects with custom light sources.
Of course, the first results were horrible. Our team stuck to the cause and tried to squeeze every last bit of information from the depth buffer. First we needed a method for deriving more depth points from the depth(disparity) map provided by the dual camera API. We algorithmically produced a new, denser map of depth points on the existing photo. Things were looking brighter, but still the visual effect of the computed lighting using the enriched depth map looked disappointing.
It was time for smoothing. We implemented different filters with various results. We needed a map of smooth contour lines that realistically follow the curves of the foreground objects. A special sauce of interpolation for enriching our map, along with some bilateral filtering for avoiding edge artefacts [sic] was the recipe that saved the day.
Armed with a high quality depth map, we were able to deduce the normal map which is fundamental for applying the lighting model of a 3D scene. Using a Phong-style lightning model, we had our first success!
At this stage, computation of depth information for a portrait photo took roughly 45 seconds, leading to very bad UX. It was time to move closer to the GPU! Our algorithm was first broken down to take advantage of multiple threads. Then all computations were rewritten for the Metal 2 SDK. Loading time dropped to around 3 seconds, a staggering improvement!
The next step was to expose all configurable parameters to the user. When our UX team started work on the project, there were dozens of parameters to tweak. That was no good, we needed a minimal set of parameters that give the user full control without being overwhelming. After lots of iterations, we narrowed down our list to six parameters: 2 global settings and 4 light source specific parameters.
Source : http://bgr.com/2018/05/21/iphone-x-camera-specs-apollo-app-portrait-lighting/