Augmented Reality in the Rhetoric and Symbolic Experience of Comic Book Images

Image narrative is an important way of expression in comics. As an art form, comics express storyline, characterization and emotion through image narrative.

Image narrative, is a narrative style that conveys information, emotions and opinions through visual elements. In a broad sense, graphic narrative has become a modern representation equivalent to visual culture, which is a basic language and expression of current culture. In a narrower sense, pictorial narratives refer to those that exist in the human cultural system and are carried by a variety of communication media. In particular, it is the narrative expression that takes image symbols such as film and television, painting, photography and advertisement as the basic ideological system. Image consists of visual and pictorial parts, and the visual part refers to photography, video, movie, television, and all kinds of advertisements and so on, which are made from real images. The pictorial part is a variety of images drawn by human beings, mainly including illustrations, comics, cartoon products, video games, etc., which together constitute today’s era of pictorial culture.

Comic symbol language includes:

Modeling symbols (character symbols, scene and prop symbols).

Textual symbols (dialog and dialog boxes, mimesis and onomatopoeia).

Icons and graphic rhetoric symbols (emoticons, speed lines and effect lines). Icons and pictorial rhetorical symbols play an important role in comics, as they can convey depth of emotion and information and help to enhance the reader’s reading experience and understanding.

Comics are a kind of graphic narrative, and images are the basic constituent units of visual text, equivalent to sentences in a language article. These “sentences” form paragraphs of an article through logical association, and sentences are connected with each other through certain rhetorical means to form a paragraph with stable structure and complete meaning. Similarly, image to image will also use a certain image rhetorical means to form a stable structure and complete meaning of the visual discourse [17-18].

The graphic rhetoric commonly used in cartoon texts contains such rhetorical patterns as imitation, metaphor, substitution, exaggeration, representation, anthropomorphism, symbolism, and quotation.

Mimicry refers to the rhetorical phenomenon of describing or recording the situation of things.

Anthropomorphism is an expressive method of endowing things with human thoughts, feelings, and activities.

In literature, simulacrum refers to portraying things that are unknowable and invisible in real life in an imaginative way as visible, audible, and vivid images. In visual images, the creation of manifestation requires bold imagination, and the objects, feelings, and situations thought of are expressed in a limited number of images, which gives people an unreal and novel visual experience. In cartoon works, the rhetorical practice of simulation is to let the audience form a figurative cognition of certain things and deepen their understanding.

In visual language, simile is manifested as seizing the similarity of two images, and imitating one thing as another. In the self-media popular science comics, the occurrence of image metaphor is often accompanied by the behavior of text metaphor, so as not to be confused by the replacement of images when reading.

In comics, the rhetorician, through imagination, subjective sensory metamorphosis of things in the objective world, exaggerates or highlights a part of the characteristics of things, in a way that breaks through the visual convention and the imaging of the physical form. Add fantasy and interesting effects to the picture to attract readers’ attention and deepen readers’ impression.

In the visual text of popular science comics, borrowing is mainly used in some concepts and events, because the concepts and events can not be expressed clearly by a simple image, so they are often replaced by an image that is logically related to the thing.

In popular science comics, visual images endowed with special symbolism appear in key positions in the picture, so that the concrete image shows abstract and profound meanings.

In recent years, the popularization of smartphones has led to the gradual transfer of people’s entertainment to cell phones. The presentation form of comics has also been gradually improved. Strip comics are mainly browsed on the mobile terminal, and their top-down layout is more suitable for mobile reading. Because of the continuity of its frame, the strip has its own unique narrative method, which has a great influence on the sub-scene of the comic. And compared with the traditional paper comics, the strips read on cell phones can make better use of people’s fragmented time, adapting to the reading habits of the modern society and bringing readers a novel reading experience.

With the continuous development of new media technology, the image narrative mode of comics is increasingly changing, and dynamic interactive comics are gradually bred as a new art mode. Dynamic comics combine the elements of comics and animation, and compared with traditional comics and strips, dynamic comics are characterized by audio-visual integration and are more interactive on mobile terminals.

Interactive comics in a broad sense is a form of digital comics built on top of dynamic digital comics, utilizing hypermedia technology and computer platforms as carriers, and combining sound and animation technology into interactive technology with traditional comics. The difference between the comics and traditional comics is that interactive comics take advantage of a specific digital platform to add animation or sound to a separate subplot or frame in a normal comic page, so that readers can freely control and participate in the process of the comic plot.

2.3.2

Augmented Reality-based Comic Support Technology

The application of augmented reality’s in interactive comics, for fans, augmented reality technology’s real-time interaction and the combination of real and virtual characteristics, so that readers experience more than just a comic book so simple, the harvest is a lot, because the AR technology brings three superimposed enhancement after the experience.

1)

Tracking registration technology

To realize the seamless integration of the virtual scene and the real scene, the augmented reality system must determine the position of the virtual object in the real scene as well as the real-time position and direction of the observer, and the tracking registration technology is the key technology to solve these problems.

This paper introduces the process of finding and matching natural feature points based on SURF algorithm, and realizes the finding and matching of natural feature points.

(1)

Feature point detection

The SURF algorithm greatly improves the performance of the algorithm by combining the fast computational nature of the integral image, which implements the fast computation of a rectangular region. Its value represents the sum of the pixel values in the rectangular image region formed by pixel X(x, y) and the origin.

For the input image I and one of the points X(x,y), the formula is expressed as follows: (1) IΣ(x)=∑i=0i≤x∑j=0j≤yI(i,j) In the above equation I_∑(x) represents the integral image value corresponding to point and X(x,y) and I(i, j) represents the pixel value of the coordinates of point (i, j) in the region.

Since the feature point finding for SURF is done through Hessian matrix determinant and at the same time the integral image is utilized to reduce the computation time. The Hessian matrix for any point X(x, y) on the image in the scale space of scale δ is defined as follows: (2) H(x,δ)=[ Lxx(x,δ)Lxy(x,δ)Lxy(x,δ)Lyy(x,δ) ] where component L_xx(x,δ) represents the convolution of the Gaussian second order partial derivatives in the x direction of point X(x,y), and the remaining components L_x,y(x,δ), L_y,y(x,δ) are similarly represented.

Combining the above formulas, the Hessian determinant can be utilized for each pixel point in the image and the values obtained are used to locate the feature points.

In order to increase the speed of computation to reduce the amount of convolutional operations and to solve the above mentioned problems, an approximate representation of the Gaussian kernel function by the box filter kernel is proposed. Thus the Hessian determinant is utilized to approximate the Gaussian kernel function accurately as follows: (3) det(Happrox)=DxxDyy−(ωDxy)2

where ω is the weight that makes det(H_approx) approximately approximate to the value of the Hessian determinant. ω The weights are calculated as follows: (4) ω=‖ Lxy(1.2) ‖F‖ Dxx(0.9) ‖F‖ Lxx(1.2) ‖F‖ Dxy(0.9) ‖F=0.912…≈0.9

|| ||_F is the Frobenius paradigm, and ω is taken to be 0.9 which has a negligible effect on the calculation.

The value of (H_approx) represents the speckle response at point X(x,y,δ) , which indicates the local maxima of the response of the point to be detected at different scales and spaces. When detecting feature points, the concept of scale space and Fast-Hessian method are combined to detect them.

(2)

Feature point matching and purification

Feature point matching is to find the feature points of the pictures that need to be aligned, and compare the feature description information of each feature point to find out the matching feature points. The more popular method of feature point matching is to utilize the trace of Hessian matrix and the Euclidean distance similarity to cooperate, so as to speed up the matching speed. Namely: (5) d(X,Y)=∑i=1N(xi−1yi)2

The error rate of the matched feature points obtained after SURF detection is small, but the mismatch is objective. The current widely used Robust Random Sampling Consistent Algorithm (RANSAC) can be used to remove the mis-matched feature points.

(3)

Image Preprocessing

The number of detected feature points is important for the efficiency and accuracy of image matching, which determines the system operation efficiency to a certain extent, so in this paper, the feature point extraction is carried out by preprocessing operations on the image first.

Usually, the histogram averaging of the gray scale map requires a mapping function to map the gray scale values to the corresponding intervals. Here, let the gray value of the original image at point (x,y) be t, the corresponding value after mapping be p, the gray level range of a map is [0,R-1], and the mapping function is Q(t). Usually as a mapping function Q(t) to satisfy two conditions:

Q(t) is a single-valued increasing function in the region of [0,R–1] . Where Q(t) is single-valued ensures the existence of an inverse transform, the incremental function ensures that the relative ordering of the gray values in the image remains unchanged after the mapping.

When 0≤t≤R–1 has 0≤Q(t)≤R–1, this ensures that the gray levels have the same range before and after the input.

Let the probability of occurrence of gray level t_k in the image be: (6) Q(tk)=nknk=0,1,2,…,R−1

Where, n denotes the sum of pixels in the image, n_k denotes the number of pixels in gray level r_k , and R is the number of gray levels in the image. Then the cumulative distribution function is represented as: (7) pk=Q(tk)=∑j=0kPr(rj)=∑j=0knjnk=0,1,2,…,R−1

Therefore, the cumulative distribution function meets the above conditions and it is possible to achieve a uniform mapping of the grayscale of the original grayscale image to the homogenized image using this function.

In this paper, bilateral filtering is chosen for denoising because Gaussian filtering and other processing images will make the edges blurred. While bilateral filtering adds the pixel intensity change to retain the edge information, so that both the denoising effect and the edge is maintained.

The formula for bilateral filtering is expressed as:

(8) B[ I ]m=1Wm∑n∈TGδd(‖ m−n ‖)Gδt(Im−In)In

where W_w means: (9) Wm=∑n∈TGδd(‖ m−n ‖)Gδt(Im−In)

G_δd is used to minimize the effect of distant pixels on the current point, and G_δt is used to represent the effect of sudden pixel changes. G_δd, G_δt both use the Euclidean distance as a parameterized Gaussian function: (10) Gδd=e−12(d(m,n)δd)2

(11) Gδt=e−12(δ(f(m),S(n))δt)2

d(m,n) represents the Euclidean distance between pixel points, δ(I(m),I(n)) the gray scale difference between pixels. The standard deviation of the Gaussian function (δ_d,δ_r) controls the bilateral filter performance by limiting the geometric spatial distance and the inter-pixel difference.

In the actual augmented reality system operation, tracking failure still occurs, so it is necessary to re-feature point detection and matching of the video image to update the single response matrix, so as to re-register the tracking.The SURF feature detection algorithm is a feature point search for all the points of the target image, due to the whole map to find and match the time required is relatively high. For the target image is very large when the feature point search is especially time-consuming, can not well meet the real-time requirements. Therefore, this paper proposes a way to optimize feature point finding by combining the position obtained from the previous frame of tracking failure. If the tracking fails in the current frame, the position of the feature point in the 2D coordinates of the image is obtained using the previous frame. It can be estimated in this regard that the image to be matched is within a certain range of this feature point position. Since the augmented reality system only needs more than four feature points to obtain the single response matrix, thus obtaining all the information needed for registration, three-dimensional registration can be realized.

2)

Real-time light rendering technology

The integration of the virtual scene and the real scene needs to realize the lighting of the two to achieve the same effect, this paper researches the real-time lighting rendering technology, respectively introduces the basic theory of local lighting model and global lighting model, and simulates the flame effect by using the particle system. A pixel enhancement algorithm is proposed, which combines the pixel enhancement algorithm and the characteristics of the flame particle system to realize the effect of global illumination, i.e., the virtual fire light illuminates the surrounding real environment.

According to the characteristics of ambient light, the following formula model is usually used to approximate the simulation of ambient lighting. Namely: (12) I=KαIα

Where K_α is the diffuse reflection coefficient of the ambient light on the surface of the object, with a value in the range of 0~1, and I_α is the ambient light intensity in the scene.

When drawing the scene, in addition to considering the effect of ambient light on the object, the role of other light sources should also be considered. According to Lambert’s law of cosines, the intensity of diffuse light reflected from an ideal diffuse reflector is proportional to the cosine of the angle between the incident light and the normal direction of the object surface, i.e.: (13) I=KdIlcosθ

Where I is the intensity of diffusely reflected light at a point on the surface of the object, K_d is the diffuse reflection coefficient or diffuse reflectance, which depends on the properties of the material on the surface of the object. I₁ is the intensity of the incident light from the point source, and θ is the angle between the incident light and the normal direction of the surface of the object, which ranges from 0° to 90°.

Phong Specular Reflection Model, an empirical formula for calculating the light intensity of specularly reflected light, or simply the Phong model, i.e: (14) I=ω(θ)I,cosnα

Where, I is the intensity of light reflected from the specular surface. I₁ for the incident light intensity, ω(θ) for the object surface specular reflection coefficient. Generally expressed in k_s , take the constant between 0 and 1. α for the direction of the specular reflection of light and the direction of the line of sight angle, take the value of the range of 0 ° ~ 90 °. n becomes the specular highlight index, determined by the smoothness of the surface material of the observed object.

Combined with the classical Phong specular reflection model Lambert diffuse reflection model to get the following Phong model with only one point of light source irradiation, that is: (15) I=KαIα+KdIlcosθ+KsIlcosnα

If there are multiple point light sources in the scene, then superimposing the illumination of each point light source gives a Phong model illuminated by multiple point light sources, ie: (16) I=KαIα+∑i=1mIli(Kdcosθi+Kscosnαi)

where m is the number of point sources contributing to the scene, I_li is the intensity of the incident light emitted by the ird point source. θ_i is the angle between the incident light from the i th point source and the normal vector of the object surface. α_i is the angle between the direction of the specular reflection from the ith point source and the line of sight direction.

In the global illumination model, the light intensity I radiated from the surface of an object in a certain direction in space is considered to consist of two parts, the reflected light intensity I_local caused by direct irradiation from the light source, and the ambient light reflected from other objects and the refracted light I_global through the object itself, i.e: (17) I=Iloal+Iglobal

Where I_local can be directly calculated by Phong local illumination model.

Using the stochastic dynamic properties of the particle system and the characteristics of the global illumination model objects reflecting light to illuminate other objects, the global illumination in the virtual scene and the real scene interaction of the lighting effect is completed.

Here, the virtual lighting is used to increase the intensity of the rendered scene. By pixel enhancement, we mean that we not only change the brightness value of the scene pixels, but also change the color value of the pixels accordingly. Using OpenGL’s programmable slice shader function, the currently stored lighting pixel values can be obtained. Then, using OpenGL’s alpha blending, the currently stored lighting pixel values are blended with the lighting pixel values of the virtual objects to form a new scene after blending.

Assuming D=(D_R,D_G,D_B,D_s) is the original pixel color value, i.e., the current stored color value from the image, S=(S_R,S_G,S_B,S_a) is the pixel color value of the new slice, i.e., the color pixel value of the flame, and then the blended pixel color value C=(C_R,C_G,C_B,C_a) is defined: (18) C=D+M×S

Where M is the material RGB α value of that pixel point, i.e. (M_R,M_G,M_B,M_o).

However for all the pixel point values of the entire captured image, it is not necessary to get a new pixel value for each point, here, the original pixel color value can be used instead, i.e.: (19) C=D+D×S

Compare the standard equation for alpha blending: (20) C=d×D+s×S

where D is the current pixel value, S is the new slice pixel value, s and d are the source and target factors, respectively, Eq. (20) d is the constant 1, and s is the current stored pixel color value or the target blending factor.

Flames are simulated using real-time light rendering, and the process of particle system simulation of flames is shown in Fig. 1. It can be summarized as generating new particles, adding different attributes and life cycles to each particle. Delete the dead particles and control the motion of the existing live particles according to the corresponding motion model. Plot all the surviving particles in the system.

Figure 1.

Flame simulation process block diagram

Observe the results of the experiment, the virtual objects (flames) that can self-illuminate illuminate the real desktop, realizing the requirement of the global illumination model, i.e., the transfer of light energy between objects. Objects under the local illumination model can generate the corresponding ambient light, scattered light or specular light according to the given illumination. The application of global illumination in augmented reality systems meets the requirements of real-time light rendering between the virtual world and the real world and increases the realism of virtual objects.

The hardware environment used in this paper is a branded laptop with Intel(R) Core(TM) i7-8250UCPU@1.60GHz1.80 GHz processor, 8GB of RAM on board, and a 64-bit Win10 operating system. The implementation is based on Visual Studio 2023 development environment, the programming language is C++, and the image processing library of OpenCV version 4.5.5 and OpenGL graphics library are used. In order to verify the performance of the improved algorithm in this paper, video sequences from the public dataset OTB100 are selected for simulation experiments of target tracking and compared and analyzed with TLD algorithm, KCF algorithm and so on.

Comparison test of this paper’s algorithm, TLD algorithm and KCF algorithm on the test platform of OTB100, the accuracy and success rate of target tracking algorithm are shown in Fig. 2. According to the curve direction of the two images, it can be seen that the accuracy and success rate of this paper’s algorithm are in the front position compared with the other two algorithms. The target tracking algorithm based on improved feature points proposed in this paper achieves an accuracy of 0.821 and a success rate of 0.813 on the OTB100 test platform.

Figure 2.

Accuracy and success rate of target tracking algorithm

The improved feature point-based target tracking algorithm is used to track the captured footage separately, and combined with the feature extraction and matching method to extract, describe and match the feature points. After that the camera’s position information is calculated. Finally, the OpenGL image library is utilized to render and draw the virtual model in the real environment and register it in the real environment. From the experimental results of virtual model tracking registration, it can be seen that the improved tracking registration in this paper has better accuracy and stability than the original algorithm’s tracking registration in the process of rotation transformation and occlusion transformation.

The software and hardware environments used in this experiment are the same as above. The experiments in this chapter will analyze the rendering performance under different voxel resolutions, mainly to verify the actual rendering effect and real-time performance of the global illumination rendering scheme proposed in this paper.

The rate at which an application displays an image is measured in frames per second (FPS), and only a little interactivity is possible when displaying one frame per second. At 6FPS, interactivity increases. And if it can display at 15 FPS, then it is a completely real-time application and the user is focused on the action and interaction.

The variation of FPS for different voxels is shown in Fig. 3, Figs. (a), (b) and (c) show the variation of FPS for 256 voxels, 512 voxels and 128 voxels, respectively.

Figure 3.

FPS changes under different voxels

Figure (a) shows the frame rate variation of 256 resolution voxels, and it can be seen that the frame rate basically stays around 40, with good real-time performance, except for the frame rate drop at the beginning when there is a need for other operations such as loading scenes. At the beginning of the 54s, the surface roughness of the objects and scenes was adjusted from the lowest to the highest. The frame rate fluctuated during the process, but after the adjustment the frame rate became higher and increased to around 43.2.

Figure (b) shows the variation of frame rate for 512 resolution voxels, and it can be seen that the frame rate decreased significantly compared to the 256 resolution experiment, basically between 15 and 20. And the roughness was similarly adjusted from 50s onwards.

Figure (c) compared with the resolution of 256, the frame rate is not very obvious, mainly stabilized at about 42.7FPS. And the adjustment of roughness from 90s onwards did not improve the performance significantly, but rather caused fluctuations.

At the same time, the experiment will adjust the voxel resolution to check the rendering effect at different resolutions. This is done by keeping other parameters constant, using the same virtual object model, and adjusting only the voxel resolution. The experimental results reflect the results of scene roughness of 0 at 128, 256, and 512 resolutions. It can be seen that the results at 512 resolution are the most detailed. 128 resolution is the roughest and has some size expansion. 512 resolution has the problem of low frame rate and is not much better compared to 256 resolution. In addition, the effect of 128 resolution has some serious deformation, and the frame rate is about the same compared to 256 resolution. In this article, 256 resolution is the best setting.

The computer required for system operation is equipped with: AMD Athlon(tm)64 4000 processor, 1G RAM, AMD Radeon HD 7950 graphics card, Windows XP operating system, and VC++6.0 environment applying DirectX9.1 for system development.

The real experience brought by the viewing process of augmented reality technology-based comics can be directly and obviously seen from the subjective feelings of the comic users, so it is necessary to design a reasonable scale. A user questionnaire is conducted, and two groups of experimental results are finally obtained and compared to draw conclusions.

In this test, the “Interactive Augmented Reality Comic User Evaluation Scale” was designed, which was designed based on the Likert scale. The Likert scale is a scoring sum scale, and each statement has five responses: “strongly agree”, “agree”, “not necessarily”, “disagree”, and “strongly disagree”, which are scored as 5, 4, 3, 2, and 1 respectively. The sum of the total scores of the same topic can indicate what kind of attitude the person is trying to hold on the issue and how strong this attitude is.

In this experiment, 50 audiences of dynamic interactive class comics were randomly selected as subjects and divided into two groups of 25 people each. The scales were filled out separately, and the results of the user evaluation of interactive augmented reality-based comics are shown in Table 2.

There are a number of topics such as “augmented reality comics can bring me a comfortable experience”, “the images of flowers and the sky generated in augmented reality comics are very realistic”, “augmented reality comics are very interactive”, “I feel natural when I use the augmented reality comics system”, “augmented reality comics give me a lot of feedback”, and “I prefer the presentation of augmented reality comics”.

Regarding the results of the user experience survey on augmented reality-based comics, the users’ experience was generally good, with a score of 4.66 on the perception of using augmented reality-based comics, recognizing the expression of augmented reality technology in modifying the images of comics.