The term “reflectance transformation imaging” (RTI) was first developed by Tom Malzbender at Hewlett Packard Laboratories, who invented the image processing methods known as polynomial texture mapping^[1]. Cultural Heritage Imaging developed the technique in the field of cultural heritage^[2,3]. The exact capture and processing pipeline have been used to survey surface information and reveal the clues that are not easily visible to the naked eye^[4–6].

Handwriting is a special kind of trace and a carrier of various information^[7]. The commonly used methods for the display of dielectric surface traces include the method of lateral light observation under a microscope, side light infrared imaging, static voltage trace development, and the method of laser confocal scanning microscopy^[8]. These methods have their own shortcomings in showing the detailed features of handwriting traces.

In our work, the application of RTI for the display of handwriting traces is studied. First, a laboratory device is set up to obtain a series of images of handwriting traces on the object. After image reconstruction of these series of images, the light direction and rendering mode can be applied interactively to obtain the detailed features of the handwriting traces. Not only can the information such as the writing content and writing order of the colored handwriting traces be obtained, but also the three-dimensional texture features of the colorless handwriting indentation. From the experimental results, the detailed features of the handwriting traces are closely related to the incident light angle and the rendering mode, especially for the colorless handwriting. Therefore, it is necessary to find an evaluation method to determine the incident light direction and rendering mode to obtain the detailed features as much as possible. An objective evaluation method without reference is proposed that can provide a basis for selecting the incident light direction and rendering mode. Furthermore, detailed features can be obtained from the images of handwriting traces.

RTI uses a set of digital photographs of a stationary object. A mathematical model describing the luminance information for each pixel in an image in terms of a function representing the direction of incident illumination was presented by Malzbender et al.^[9]. The x and y coordinates of the projection of the normalized light vector onto the image plane are given by the light direction (lu, lv). The luminance function for each pixel is approximated by a biquadratic polynomial in lu and lv: L(u,v;lu,lv)=a0(u,v)lu2+a1(u,v)lv2+a2(u,v)lulv+a3(u,v)lu+a4(u,v)lv+a5(u,v).(1)

The six coefficients a₀–a₅ are stored with the unscaled RGB (red/green/blue) values of the pixel. By making a series of images with the camera and specimen in fixed positions, and with varying directions of incoming light, the coefficients of the luminance polynomial for each pixel can be computed by a least-squares method (regression) using singular value decomposition. The normal vector per pixel can be estimated by computing the brightest incident illumination^[10].

The whole set of normal vectors provides the “description” of the topography of the object accurately and completely. So, reproducing pixel by pixel the surface texture as well as its color and reflective properties to create a digital map is possible. The properties including surface interreflection, subsurface scattering, and self-shadowing can be recorded in the per pixel information. It is possible to independently adjust the reflectance or color values to enhance the contrast between high and low reliefs or reveal details about surface irregularities. Consequently, RTI can be understood as two-dimensional images possessing true three-dimensional data^[11].

The color and shape data of an object can be obtained through these photographs, and a digital map of the object surface can be further synthesized by using RTIBuilder. The user can view the generated digital map through the RTI visualization program called RTIViewer. When the user moves the virtual light source in the software, the digital model can reproduce the reflection model of the real object and realize the perception of the target in three-dimensional space^[12]. Different lighting directions can be interactively implemented, and the contrast of surface details can be amplified by simulating large amounts of inclined light^[13–15].

In order to study the application of RTI for the display of handwriting traces^[16–18], the plain A4 paper materials with handwriting traces are chosen (Fig. 1). There are some colored handwriting traces, as shown in the rectangle on the lower right of Fig. 1, and some colorless handwriting indentations which we can hardly identify, as shown in the rectangle on the upper left of Fig. 1. The BrukerStylus Profiler (Dektak XT) was applied first to measure the depth of local traces at the two small rectangles I and II in Fig. 1. Figure 2(a) shows the measured results of the trace at I and Fig. 2(b) shows the measured depth data of the trace at II in Fig. 1. According to the measurement results, the depth of the trace at I is about 22 μm, and the depth of the trace at II is about 13 μm. These will provide a quantitative reference for the results of RTI.

The simple capture device is shown in Fig. 3(a). A Canon digital SLR camera (EOS 60D) with a Canon EF 100 mm f/2.8 Macro IS USM lens was fixed on the camera’s auxiliary support directly above each object, and the camera was aligned to the object. We captured images through the EOS utility software on the computer. The CREE XLamp MC-E LED was placed in the designated positions in space in turn, forming a virtual domed light. The radius of the virtual light dome (between the LED and the object center) is usually three times the length of the object’s longest edge. So, the RTI data was obtained from a virtual dome with a radius of about 50 cm. A total of 58 different light source positions constituted our virtual light dome. The latitudes were 15 deg, 30 deg, 45 deg, 60 deg, and 75 deg. The light sources on each horizontal plane were placed equidistantly, and the adjacent light sources were 0.189 m arc length apart. So, at every latitude, there were 16, 16, 12, 9, and 5 different light source positions, respectively. The top view of the light source position in the virtual dome is shown in Fig. 3(b), from which the light source can be seen to be evenly distributed. A dark, shiny ball, which was used to calculate the direction of light, was placed near the sample but not hidden from shadow. A series of 58 raw images of the object in different illumination directions were able to be gotten. RTIBuilder could be used to detect the direction of lights and convert multiple images into a single PTM file. The RTIViewer software can be used to observe the PTM file, including changing the mode of visual rendering and the direction of light to enhance shape perception, so as to observe the detailed features of the handwriting traces^[19].

The colored handwriting traces and colorless handwriting indentations on the surface of the object were analyzed by RTI. We first determine the rendering mode and then interactively adjust the illumination direction to get the images that have the better detailed features of the handwriting traces.

The results of colored handwriting traces applying RTI are shown in Fig. 4. The position of the light source in Fig. 4(b) is more inclined than in Fig. 4(a). In this mode, the written content through the two images can be gotten easily. In addition, the writing characteristics of some handwriting traces, such as the stroke order, also can be gotten. Two images with better detailed features under the specular enhancement mode could be obtained, as shown in Figs. 4(c) and 4(d). The specular enhancement mode is particularly effective when enhancing the perception of surface shape. By observing the overlapping characteristics of “迹”, the order of the handwriting can be inferred.

The results of colorless handwriting indentations applying RTI are shown in Fig. 5. Figures 5(a) and 5(b) are the result images in default mode. Through these two images, the parts of the writing content of the handwriting indentations can be obtained. The characteristics of the penmanship of the writer and other information also can be inferred by obtaining the three-dimensional shape of the handwriting indentations. Since there is no obvious color information of the handwriting indentations, and the object background is white, the visualization effect of the handwriting indentations is not obvious in this mode.

Figures 5(c) and 5(d) are the result images in specular enhancement mode, and the light source position in Fig. 5(d) is more inclined. In this mode, the difficulty in recognition could be largely overcome because of the enhancement of the perception of the surface shape. The three-dimensional impression of the handwriting indentations can be observed in Figs. 5(c) and 5(d). For example, through the number “2”, the calligraphy features can be obtained clearly. The more inclined the light would be, whether it would be in default mode or in specular enhancement mode, the easier it would be for the colorless handwriting to appear, so it would give more detailed features. In general, the details of Figs. 5(c) and 5(d) are more legible than that of Figs. 5(a) and 5(b). This shows that for a colorless handwriting indentation, the detailed features will show up better in specular enhancement mode.

The images shown in Figs. 4 and 5 are not from the position of the same light source. Under the same conditions, the more inclined the light direction is, the more obvious the subtle features of the handwriting trace will be through the effect of shadows and so on. In the default mode, the diffuse color percentage is 100% and the highlight percentage is 0. In this mode, only diffuse RGB color information is rendered, and the light direction can be inclined as much as possible to get a relatively clear image, subjectively. In the specular enhancement mode, the highlight is 100%, while the diffuse reflection component is 0. Therefore, the appropriate “inclined” light direction can be found in this mode to obtain better trace characteristics. If the same light source location is selected in this mode as in default mode, the image surface is black and no trace can be observed. The best viewpoint from the subjective evaluation in each of the modes is different for the characteristics of the different modes.

Through the application of RTI for the display of handwriting traces, especially for colorless handwriting traces, the detailed features that cannot be observed easily by the naked eye can be distinguished. Through Fig. 5(d), the local detailed features of regions I and II can be clearly obtained. Compared with the previous measurement results of BrukerStylus Profiler, the display range of RTI in the handwriting traces can be as small as 13 μm. For the characteristics of handwriting traces, applying the specular enhancement mode and oblique incidence illumination can get good information such as the calligraphy features and writing styles. In the specific identification, the information such as the characteristics of writing content and writing style, can reveal the facts of the cases and clarify the nature of the cases.

As the above experiment results show, RTI can be applied well in the field of handwriting traces. Although the subjective evaluation (human eyes) is simple, intuitive, and operable, the subjective factors are introduced into the results. The experimental results may be changed due to the change of the people in the evaluation. Therefore, an objective, non-referential evaluation method based on the image results of the application of RTI can be proposed.

Since a clear image contains more detailed information than a fuzzy one, namely the high-frequency component, the image resolution can be evaluated by measuring the amount of high-frequency information contained in the image^[20]. In combination with the imaging model of optical systems, a method named no-reference structural sharpness (NRSS) is applied to evaluate the high-frequency component^[21,22].

Structural similarity (SSIM) is a full reference image quality evaluation method^[22]. A full reference image quality evaluation method of SSIM for image blocks x and y, contains three parts: brightness comparison, contrast comparison, and structural information comparison, which are defined as $$\begin{gathered} Brightness:L(x,y)=2μxμy+C1μx2+μy2+C1, \\ Contrast:C(x,y)=2σxσy+C2σx2+σy2+C2, \\ Structuralinformation:S(x,y)=σxy+C3σxσy+C3, \end{gathered}$$(2)SSIM(x,y)=[L(x,y)]α[C(x,y)]β[S(x,y)]γ,(3)where α, β, γ>0, and the three parameters are weighted for brightness, contrast, and structural information.

The steps of NRSS are as follows.1)Construct a reference image for the image to be evaluated. Define the image to be evaluated as I, and then the reference image Ir is defined as Ir=LPF(I).(4)In the experiment, we applied the Gaussian smoothing filter, which has a size of 7×7 and σ2=6.2)The gradient information of the reference image Ir and image I was extracted. Since the standard for humans to judge whether the image is clear mainly comes from the edge and contour of the image, we use a Sobel operator to obtain the image gradient information to extract the edge information of the image. The gradient information of I and Ir is defined as G and Gr, respectively.3)Find N image blocks with the most abundant gradient information in gradient image G. The image G is divided into small blocks of 8×8, and the step size between blocks is 4; that is, the adjacent blocks have 50% overlap in order to avoid losing important edges. N=64 is set in this experiment.4)Calculate the NRSS of the original image without reference. First, the structural similarity SSIM(xi, yi) of each xi and yi was calculated, and then the unreferenced structural clarity of the image was defined as NRSS=1−1N∑i=1NSSIM(xi,yi).(5)

That is, the value of NRSS is between 0 and 1. The closer the NRSS value is to 0, the less the high-frequency information of the image is; that is, the image quality is relatively poor. On the contrary, if the NRSS value is closer to 1, it indicates that the image has more high-frequency information; that is, the image quality is relatively good.

The objective evaluation method was applied to the RTI results of two types of handwriting traces. As shown in Fig. 6, the digitized results are obtained by applying the objective evaluation method to the images of the colored handwriting traces. Among them, Fig. 6(a) (NRSS: 0.013743) and Fig. 6(b) (NRSS: 0.015857) are images in the default mode. It is observed that the handwriting traces in the two images show similar detailed features, but the details of Fig. 6(b) are relatively more abundant. Accordingly, the objective evaluation results in Fig. 6(b) are slightly better than that in Fig. 6(a), which is consistent with the subjective evaluation results. Figure 6(c) (NRSS: 0.042854) and Fig. 6(d) (NRSS: 0.054382) are the images obtained while the specular enhancement mode is used. The detailed features of the two images on the handwriting traces are significantly better than those of Figs. 6(a) and 6(b), and the corresponding values of NRSS are also greater than that of Figs. 6(a) and 6(b).

As shown in Fig. 7, the digitized results are obtained by applying the objective evaluation method to the images of the colorless handwriting indentations. Figure 7(a) (NRSS: 0.069636) shows the image of handwriting indentations in the default mode. Figure 7(b) (NRSS: 0.082468) and Fig. 7(c) (NRSS: 0.087245) show the images in the specular enhancement mode. Figures 7(b) and 7(c) in the specular enhancement mode show that detailed features are significantly better than those in the default mode, and the corresponding NRSS values are also greater than that in the default mode. Figures 7(b) and 7(c) have similar subjective results and NRSS values. Especially for colorless handwriting traces, which are very weak, it is difficult to distinguish the differences under various rendering modes and lighting directions subjectively. Therefore, this objective evaluation method can be applied to digitize the detailed features.

It also provides related experiments for discussion in order to validate the effectiveness of the objective evaluation method, NRSS. Figure 8 shows the colored handwriting traces, which are written by fountain pens and gel pens, respectively.

Figure 9 shows a colorless handwriting trace, which is shallow and hard to be obtained by subjective observation.

This objective evaluation method can objectively evaluate the handwriting traces in which the detailed features are the most abundant in the light direction and rendering mode. Subjective evaluation (human eyes) of two images with similar effects has similar objective evaluation results. The two images whose subjective evaluation results differ greatly also differ greatly in objective evaluation results. The results of this objective evaluation method are consistent with that of the subjective evaluation method, which is more versatile, correct, and objective.

The application of RTI for the display of both colored and colorless handwriting traces was discussed, respectively. Both the two-dimensional and three-dimensional texture features of handwriting traces can be obtained. Especially for colorless handwriting indentation, the detailed features were obtained and illustrated, which is of great significance for the display of handwriting traces. In order to evaluate the results of RTI oblique illumination and visual rendering, an objective evaluation method without reference was accounted for and used. The effect of the objective evaluation method is consistent with the subjective, namely naked eyes, results. This evaluation method proves to be effective to obtain more details and expand the application of RTI in criminal investigations and other fields.