Hyperspectral imaging is widely used in various fields such as environmental observation, medical diagnosis, agricultural, waste sorting or food quality control [1–6]. Most of the used hyperspectral cameras use scanning approaches (spatial, spectral or mixed) to capture the full data cube consisting of a two-dimensional image for each spectral channel [7, 8]. This leads to problems such as motion artifacts or an increased mechanical complexity (moving parts). The computed tomography imaging spectrometer (CTIS), in contrast, uses a non-scanning approach. It belongs, such as the pixel-level filter array camera or the multiaperture filtered camera, to the snapshot spectral imaging cameras [9, 10].

A CTIS system computes the hyperspectral information from a single captured sensor image. A typical optical design is illustrated in Figure 1a. A field stop is used to limit the size of the imaged scene. It is usually placed in the intermediate image plane of the first lens. A subsequent diffractive optical element (DOE) creates a spatio-spectral smeared two-dimensional sensor image. The zeroth diffraction order is usually located in the middle of the sensor and is equivalent to a panchromatic image of the scene. Multiple higher diffraction orders, that are similar to oblique projections of the hyperspectral data cube, are arranged around it.

A reconstruction algorithm is required to derive the hyperspectral image from the sensor image. It solves a similar problem as the ones used for computed tomography scanners. Multiple two-dimensional projections are used to infer a three-dimensional data cube. In this paper, we use an implementation based on the expectation-maximization (EM) algorithm that utilizes spatial shift-invariance [11]. It is a model based Maximum-Likelihood iterative solver.

The system model is usually derived from a measured point spread fuction (PSF) for every spectral channel to be reconstructed. The PSF is a commonly utilized metric to assess the quality of imaging systems by measuring the image of a point light source. For a CTIS system, the resulting images look different from normal, as they include not only the direct image of the point light source at the center (zeroth diffraction order), but also the diffracted ones surrounding it. This point pattern still includes the optical aberrations, but more important, it also includes the position and spectral displacement of the higher diffraction orders. With increasing wavelength, the points move farther away from the zeroth diffraction order.

The PSF has to be measured only for one central point light source because spatial shift-invariance is utilized. In a perfect system, the sensor image of a scene matches the spectral sum of all PSF convolved with the corresponding (ground truth) spectral images of the object. The algorithm optimizes the estimated hyperspectral image iteratively by minimizing the difference between the computed sensor image and the measured sensor image.

The EM-algorithm is very sensitive to violations of the spatial shift-invariance. The reconstruction quality decreases considerably when this requirement is not met. Possible reasons for this are, for example, distortion in the sensor signal, spatially dependent aberrations, or noise in the PSF and the measured image of the scene. We use several pre-processing steps to optimize the match between the PSF and the pre-processed sensor signal, and hereby the reconstruction quality.

Furthermore, the EM-algorithm is very computationally intensive, which can lead to reconstruction times of several minutes. In recent years, there has been an increasing number of research efforts aimed at solving this problem by using neural networks [12–15]. We expect our system to not only decrease in reconstruction time, but also to increase in reconstruction quality when this approach is used.

One major drawback we want to address is the relatively big size of most published CTIS systems [16–18], which often hinders their use outside of laboratory environments. The main reason for this is that the focal length of the system has to be small when a reasonably large field of view (FOV) is captured (full scene has the size of the zero order projection in the image plane). The minimal focal length of the re-imaging lens is limited because a distortion arises when the angles of the diffracted rays become large. The effect is caused by the non-linear diffraction of light in image space (so called “conical diffraction”). Harvey et al. shows how this can be calculated in direction cosine space [19]. The first two lenses, therefore, act as a beam expander. They reduce the ray angles and hence the total focal length. The typically used Keplerian type of beam expander has the advantage that the intermediate image can be used to limit the imaged scene (field stop). However, we have found that in some cases it is beneficial to use a Galilean type of beam expander instead (Fig. 1b). It often achieves a better optical performance in a smaller form factor when the same complexity of lenses is used. The field stop is placed with some distance in front of the first lens. One major drawback of this approach is that the reconstructed hyperspectral image is considerably vignetted if the field stop is placed close to the system or a small f-number is used. The optimal design depends on the requirements of the system.

The goal for the prototype was to develop a system with a very compact form factor. It should also have a large FOV and a high resolution. To achieve this, we allow a certain degree of vignetting. All optics, except the DOE, should be ready-made. The system is designed for visible light, as there is a larger selection of off-the-shelf components for this range.

After evaluating several lens combinations in an optical design program, we decided on the design shown in Figure 2. It is based on the Galilean beam expander. The first element is the field stop that is made of laser cut cardboard. It is followed by an absorptive longpass filter (Hoya Y48) that is used to block light below 480 nm. Lens 1 is realized using two stacked lenses with a focal length of −4 mm each (SLM-04B-04N from OptoSigma). The lenses are anti reflection coated, and we additionally used a thin black marker to draw apertures directly onto them. This helps to reduce stray light. Lens 2 is an achromatic lens with a focal length of 7.5 mm (#49-275-INK from Edmund Optics). Together, those lenses achieve a beam expansion of factor 4.4 and consequently an angle reduction by roughly the same value. Subsequent to the beam expander is a dichroic shortpass filter (#47-817 from Edmund Optics) with a cut-off wavelength of 750 nm. It cannot be placed behind the longpass filter because it has an angular dependent transmission spectra. We located it behind the beam expander, where the ray angles are lower.

It is important to spectrally limit the light transmitted by the system. Light with a wavelength for which there is no PSF measurement disrupts the reconstruction. The resulting signal cannot be assigned to any reconstructed spectral channel by the algorithm.

The following DOE is a custom, in-house produced binary computer generated hologram (CGH). It is made of photoresist on a 500 μm thick glass. It creates a 5 × 5 arrangement of the projections. Its design has been computed using Fourier optics and the binary search algorithm with a loss function that is optimized for CTIS [20]. The goal is to have an equal irradiance of all projections on the sensor. The diffraction orders that are farther away from the optical axis are spread over a bigger area. Therefore, they need more light. This approach does not work for the zeroth diffraction order because the height of the binary structure can only be optimized for one wavelength. The more the wavelength of the light deviates from the optimum, the worse the diffraction efficiency and the stronger the zeroth order. These two effects result in a zero order projection which is around 12 times brighter than the higher order projections. For a good signal-to-noise ratio of the higher order projections, we always need to acquire a second, overexposed image with a higher exposure time. This way we obtain an image with a well exposed zero order projection and one with well exposed higher order projections.

The prototype is therefore not single shot capable. There are different possibilities to achieve a real single shot system. It can, for example, be accomplished with a small absorptive filter placed directly in the center of the image sensor. Something similar has been done by Okamoto et al. in one of the first CTIS publications [21]. They use an attenuator in front of an additional image plane, which is imaged using a camera. Another possibility to avoid the need of capturing two images with different exposure times is to use a DOE that is optimized for a broad spectral range [22].

Another optimization goal for the CGH design is to have a low stray light level. Because the zeroth order is much brighter, every stray light overlapping the higher order projections significantly worsens the signal, resulting in a reduced reconstruction quality. We cannot perfectly eliminate the stray light. Therefore, we reduce its influence by a pre-processing step as explained in Section 4. Figure 3 shows a part of the design file and a photo of the glass waver with several holographic structures next to each other. The CGH is glued directly onto the holder of the smartphone lens with viscous UV-curing glue. The pixel sizes in horizontal and vertical direction are determined by the maximum spatial frequencies in horizontal and vertical direction. Each spatial frequency corresponds to one spot of the spot pattern generated by the hologram. In the chosen geometry this results in pixel sizes of 0.95 μm and 1.3 μm. The manufacturing goal for the height of the positive pixels is 442 nm. With our photoresist this gives the best diffraction efficiency for light with a wavelength of 575 nm. The phase difference for this central wavelength is 180°.

The lens from a Sony Xperia 10 Plus smartphone (illustrated as a black box model) is used as re-imaging lens. We chose it because its stop is close to the front of the lens housing and its large image size (compared to the focal length of approximately 4 mm). The stop is used as the stop of the full system. Therefore, the light travels through the smartphone lens in the intended way. We also fix it to the aluminum mount with glue. The See3CAM CU135M from e-con Systems, with a monochrome, 10 bit, 13 MP image sensor, is used as the camera body. It has a pixel size of only 1.1 μm. The vital parts of the mechanics are made of anodized aluminum, everything else is 3D printed.

A photo of all the individual components and the assembled prototype can be seen in Figure 4. The total optical track length is only 21; the total size including the housing is 36 mm × 40.5 mm × 52.8 mm. The effective focal length of 0.9 results in a diagonal fov of 29°.

Figures 5a and 5b show raw sensor signals taken of a ColorChecker placed at a distance of 500 mm. We used a 150 W halogen floodlight, resulting in exposure times of 62.5 ms for the zero order projection and 1 s for the higher order projections. These long exposure times can be reduced by using a brighter light source. In our case, this was not possible because shorter exposure times caused problems due to a rolling shutter artifact originating from the flickering of the AC-powered floodlights. The problem can be solved by using DC-powered lights. To reduce the noise level of our system for this demonstration, we always acquire 16 images and sum them. The system model (PSF for every wavelength) is measured using a fiber-coupled monochromator with a bandwidth of 7 nm. The output fiber was placed centered in object space.

Some pre-processing is required to achieve the best result with the EM-algorithm. The following steps are usually performed for each measurement:

subtraction of pre-captured dark images,

The stray light removal gives the greatest improvement in reconstruction quality. The stray light originates mainly from the CGH and cannot be completely avoided. We accomplish the removal in two steps: A first stray light image is computed by convolving the cropped zero order image (low exposure time) with the zeroth diffraction order of the summed PSF (high exposure time). This approximates the stray light near the center, which is well described by the PSF. The second stray light image corrects the stray light farther away from the center. It is computed using a two-dimensional polynomial fit of order 3. Multiple sampling points are picked manually that are then used to calculate the image. The points are chosen to be in areas outside the projections. The signal would be zero here for a perfect system. An example of the combined stray light image is shown as a false color image in Figure 5c. It is subtracted from the high exposure time sensor image. The stray light accounts for up to 20% of the signal around the higher diffraction orders. The low exposure time image used for the zero order projection does not have a significant amount of noise and does not need to be corrected. The final pre-processed image is shown in Figure 5d.

Furthermore, we perform similar pre-processing steps with the measured PSF. To reduce noise, and the influence of local invariance, we perform a two-dimensional Gaussian fit for each spot and replace it with the result. This gives a symmetric, noise free spot. An RGB image, calculated from the PSF’s pre-processed hyperspectral data cube, is shown in Figure 6. Since the images are very sparse, it is strongly overexposed to increase the visual spot size. One spot from the bottom left projection for 670 nm is depicted and shown in grayscale in the original and fitted version.

Several factors influence the spectral characteristics of the reconstructed hyperspectral image. This includes, for example, the spectral distribution of the halogen light sources and the monochromator, and the spectral sensitivity of the optics (including the DOE) and the sensor. To obtain results comparable to ground truth measurements, we spectrally calibrate the reconstructed hyperspectral image. We use the reconstructed measurement of a Spectralon reflectance standard to determine a calibration curve. This curve contains a correction factor for each spectral channel. The Spectralon standard has a reflectance value of almost 100% over the entire used spectral range. Our measured spectrum gives the difference to this. The calibration curve has to be determined only once for a system when the illumination is kept constant.

The reconstruction result of the measurement taken of the ColorChecker is illustrated in its RGB representation together with some selected spectra in Figure 7. It has a yellow tint because no blue light is captured. The reconstructed hyperspectral image has a spatial resolution of 400 × 300 pixel with 39 spectral channels (7 nm steps from 488 nm to 754 nm). The result is in a good agreement with the ground truth values. We assume that remaining inaccuracies originate from noise, imperfect stray light correction, and artifacts of the reconstruction algorithm. A custom lens design and a DOE optimized for a broader spectral range together with an improved reconstruction (for example by using neural networks) could help to improve the performance even further.

The overall system, including the reconstruction, is not linear. Therefore, the quality of the reconstruction is strongly object-dependent. We reconstructed with a step size of 7 nm because it gives good results, and it is the bandwidth of the monochromator. The PSF measurement can thus be used directly for the reconstruction. For simpler scenes, it is possible to reconstruct with a higher spectral resolution without quality loss. Simple to reconstruct scenes are, for example, very sparse (small object in front of a black background). Depending on the application, a tradeoff can be made between spectral resolution and reconstruction quality.

A strongly miniaturized and portable CTIS system has been presented. Despite its size, it has a large resolution and field of view. We achieved this using a Galilean instead of a Keplerian beam expander. It can be used in applications where a small physical size is required. The CGH has been optimized for the use with CTIS systems. Several pre- and post-processing steps are necessary to achieve a good reconstruction quality. The final result is in a good agreement with the ground truth values.