Three-dimensional (3D) acquisition has been the most important technique for augmented reality or virtual reality industries. Stereo vision^[1–9] can obtain 3D information of objects using disparity between two images captured by a stereo camera. However, due to lack of perspectives for 3D objects in this technique, accurate 3D information may not be generated and the number of viewing points is limited. By contrast, multi-view or super multi-view^[10–24] imaging techniques can obtain more accurate 3D information because they use multiple cameras for pickup of 3D objects so that more perspectives and viewing points of 3D objects can be recorded. However, these conventional 3D imaging techniques have complicated system structures (e.g., synchronization among cameras) and require a lot of cameras. To overcome these drawbacks of conventional 3D imaging methods, axially distributed image sensing (ADS) has been proposed^[25–28]. It has a very simple system structure and requires only a single camera. It moves the camera along with its optical axis to record slightly different perspectives of 3D objects.

In typical 3D imaging techniques such as multi-view imaging, super multi-view imaging, or integral imaging, the reconstructed 3D image at the arbitrary viewing point (i.e., free view reconstruction) can be generated since multiple images captured by multiple cameras have different perspectives^[29,30]. On the other hand, in ADS, it may be difficult to obtain free view images due to lack of the large perspectives for 3D objects. Therefore, in this Letter, we propose a free view reconstruction method in the ADS technique. Our method utilizes elemental images with slightly different perspectives by ADS and sets a virtual camera at the arbitrary position for free view reconstruction. To support our proposed method, we show the experimental results.

The remainder of this Letter is organized as follows. First, we overview the pickup and reconstruction processes of ADS, present free view reconstruction of ADS, and discuss its limitations. Then, to support our proposed method, we show the experimental results. Finally, we conclude with summary.

Free view reconstructions for integral imaging by tilting elemental images and the reconstruction plane have been previously reported^[29,30]. Since more perspectives of elemental images can be generated in integral imaging, it is easy to implement free view reconstruction^[29,30]. However, in ADS, it may be difficult to obtain the reconstructed images at any viewing point due to lack of perspectives in elemental images. Therefore, in this Letter, we employ the virtual viewing camera for free view reconstruction of ADS. We set the position of the virtual viewing camera arbitrarily and then find the matching pixels from the elemental images captured by ADS.

Figure 1 illustrates the basic concept of the conventional ADS^[26]. This technique has a very simple structure (e.g., one-dimensional movement of the camera) and can obtain the multi perspectives of 3D objects. In addition, since the resolution of each elemental image is the same as the camera, high-resolution reconstructed 3D images can be provided.

Moving the single camera along with its optical axis, as depicted in Fig. 1, the elemental images are recorded with slightly different perspectives and magnification ratios. Using these features of ADS, the reconstructed 3D images at the front viewing point can be obtained, as shown in Fig. 2(a).

The reconstruction process is based on the inverse ray projection model, as shown in Fig. 2(b)^[26]. We suppose that the reconstruction plane of the 3D image (P) is at distance z. For a fixed z, the reconstruction planes are generated within the field of view of the Nth camera, as shown in Fig. 2(a). Thus, we can consider the Nth camera as the virtual viewing camera for the reconstruction planes. In addition, an image pixel on each reconstruction plane is projected back to the elemental image through each virtual pinhole. The inversely projected image pixel position is demagnified according to the distance z. Therefore, for N cameras, the reconstruction process can be implemented by P(x,y,z)=1N∑n=1Npn(−x/Mn,−y/Mn),(1)where Mn=zn/g=[z−(n−1)Δ−z1]/g is the magnification ratio for the nth elemental image, Δ is the distance between two adjacent cameras, g is the gap between the imaging lens and the image sensor, pn is the matching pixel in the nth elemental image, and N is the total number of elemental images, respectively.

In fact, free view reconstruction can help 3D object recognition to be more accurate because it has more information of the 3D object than the conventional reconstruction. However, in ADS, it is not easy to obtain more information of 3D objects because of low perspectives from elemental images. Therefore, in this Letter, we employ the virtual viewing camera to produce more 3D information in ADS.

Figure 3 shows the principle of free view reconstruction for ADS using the virtual viewing camera. In our proposed method, we first define the Nth camera as the virtual viewing camera, as shown in Fig. 3. The virtual viewing camera can be shifted and rotated to the desired position. When the camera is moved, the reconstruction planes are also moved with the same parameters. For a simple calculation, we consider the movement of both the virtual viewing camera and the reconstruction planes. Here, the rotation and the shift matrix operators are [XYZ]=R[xyz]−T,(2)R=[1000cosθx−sinθx0sinθxcosθx][cosθy0sinθy010−sinθy0cosθy][cosθz−sinθz0sinθzcosθz0001],(3)T=[sxsysz],(4)where X, Y, Z are the destination position of the desired reconstruction plane in Cartesian coordinates, R is the rotation matrix, T is the translation matrix, θx, θy, θz are the rotation angles in Cartesian coordinates, and sx, sy, sz are the shifting distance from the front view position, respectively.

After moving them, free view reconstruction of ADS, as shown in Fig. 3, can be implemented by finding the matching pixels between the image pixel of the moved reconstruction plane and each elemental image. The final image by free view reconstruction is generated by summation of all matched elemental images. This is given by F(X,Y,Z)=1N∑n=1Nrn(−X/Mn′,−Y/Mn′),(5)where Mn′=Zn/g=[Z−(n−1)Δ−Z1]/g is the magnification ratio of the virtual viewing camera and rn are the matching pixels between the virtual viewing camera and the nth elemental image.

To calculate the parallax between the free view reconstructed images, we used the distance of the center point between them (i.e., disparity) as D(θ1,θ2,zr)=zr(sin|θ1|+sin|θ2|),(6)where θ1, θ2 are two free view angles for reconstruction, zr is the reconstruction depth, and D is the disparity between two free view reconstructed images, respectively.

To demonstrate our method, we carry out the optical experiment. To capture the elemental images, we use a commercial digital camera (Nikon D3200) with a focal length f=50mm. It has 2184(H)×1426(V) pixels and it is moved by uniform step Δz=1mm. We capture 170 elemental images to obtain the perspectives for 3D objects, as shown in Fig. 4. 3D objects are two sign boards with “NO PARKING” characters and a toy car with “6” character. Two sign boards are tilted by −20° and 20° from the optical axis, respectively. All 3D objects are located at 500 mm from the camera. In this experiment, the parallax between two free view reconstructed images with 20° left and right views can be calculated as the disparity. Thus, the parallax D=342.0201mm.

Figure 5 shows the reconstruction results with various reconstruction depths using conventional ADS reconstruction (i.e., the front view). The focused image was obtained when the reconstruction plane was at z=500mm, which is the same as the original position of the 3D object (toy car). Otherwise, the two sign boards were blurred.

Next, we moved the virtual viewing camera to the position as shown in Fig. 6. We generated the plane images according to distance z. Examples are shown in the right of Fig. 6. We can generate the right reconstructed image at the distance z=300mm. The objects were reconstructed in the center of the plane image. This implies that the proposed free view reconstruction method is more effective.

In addition, the virtual viewing camera was moved and rotated as shown in Fig. 7. To do this, we can observe different perspectives of 3D objects. As shown in Fig. 7(b), only the left sign board is focused and otherwise is defocused in the free view reconstruction result with a tilting angle of −20°. Using free view reconstruction with a tilting angle of 20°, we can observe that the right sign board is focused, as shown in Fig. 7(c). From the result of Fig. 7, we prove that the proposed free view reconstruction method can generate plane images at any position and any angle of virtual viewing camera.

We propose a 3D free view reconstruction technique for ADS. In conventional ADS, only a front view reconstructed image can be obtained but the reconstructed 3D images at any viewing point can be obtained in our proposed method. Using our proposed method, more information can be achieved for 3D objects so it can be applied to various 3D applications. In future works, we will investigate solutions to solve the limitation of the viewing angle and depth range in ADS.