Lytro 3D visualization

The Lytro Camera captures and stores a light field instead of a standard two dimensional focused image. The light field can be processed to generate a regular image, and this image can be focused after the fact (or you could simulate a pinhole lens and have everything in focus). You can also extract depth information from the light field.

Via Twitter, I found Niriv Patel's discussion of the Lytro LFP file format and lfpsplitter extractor program and grabbed some of the sample LFP files from Lytro's website. The LFP files I found contained 7 JPEG images each focused at a different depth, a 25x25 depth field for determining which JPEG to display when the image is clicked, and some extra metadata.

I wrote an OpenGL program that creates a texture from the depth field and then uses it as an alpha mask to draw the focused region of each JPEG in the LFP archive. Here's the fragment shader:

uniform sampler2D tex;
uniform sampler2D depthtex;
uniform float startdepth;
uniform float enddepth;
varying vec2 v_texcoord;
void main() {
    float sampledDepth = texture2D(depthtex, v_texcoord).r;
    if (sampledDepth < startdepth) sampledDepth = 0;
    else if (sampledDepth > enddepth) sampledDepth = 0;
    else sampledDepth = 1.0;
    gl_FragColor = texture2D(tex, v_texcoord) * sampledDepth;
}

I had some other variations which had antialiasing to the edges of the region, but it didn't improve the output noticeably. You can see that the resultant image slice has some odd contours to it as a result of the bilinear interpolation of the low resolution depth texture. There is also a huge gradient around the edge of the butterfly, and the interpolation there gives the butterfly multiple sets of wings.

With each of the layers separated out, I could arrange them appropriately in Z, apply a perspective transform and pan them around. Then I realized that I could bring the rendering to a web page by exporting each layer as a PNG and using CSS 3's 3D Transforms module to apply the Z and perspective transformations. This is what is running above (if you're viewing in Safari or Chrome with an appropriate GPU).

Once I had this working, I wrote (5 lines of!) JavaScript to respond to accelerometer events to give the illusion that the 3D stack is not moving relative to the viewer, like a hologram. Because it uses the accelerometer (fewer devices have a gyroscope) it only works if you hold the device facing the ceiling.

Here's a video it in action on an iPad 1:

Better quality output could be generated with a higher resolution depth field. Then you could have accurate edges for the layers, and could maybe use the depth field to apply bump mapping (and some day iOS will support WebGL outside of iAds and we'll be able to use it there, too). Lytro have demonstrated a 3D illusion rendered via an animated GIF. It looks like there is perspective along the body of the camera in that photo, so it's not just parallax layers, but presumably it can't move further than the distance of the sensor without revealing "shadows".