ArticlesAll Issue
ArticlesMosaic Stylization Using Andamento
• Dongwann Kang1 and Sanghyun Seo2,*

Human-centric Computing and Information Sciences volume 12, Article number: 09 (2022)
https://doi.org/10.22967/HCIS.2022.12.009

Abstract

In this paper, a study on Andamento, which is a traditional mosaic style, is described and a method for simulating a tile mosaic with various types of Andamento is presented. We classify Andamento according to the characteristics of the tile and represent the characteristics as energy terms. We introduce a method for generating a mosaic of a certain Andamento by minimizing a mosaic energy function composed of a weighted sum of the energy terms. We propose an iterative algorithm for optimizing the energy function, and the algorithm consists of global refinement, which is a heuristic approach, and local refinement optimized by employing an evaluation of the energy function. The global refinement process causes the positions of the tiles to converge quickly toward the direction estimated to reduce the energy function. A local refinement elects the candidates of the edge transform of the tile that are expected to decrease the energy function and then evaluates such function. It optimizes the energy function by selecting an edge transform that reduces the energy function most significantly and applies the transformation. Finally, we introduce some post-processes that add texture effects of a tile and grout space.

Keywords

Non-photorealistic Rendering, Stylization, Mosaic, Andamento

Introduction

A mosaic, which is one of the interesting research topics of non-photorealistic rendering (NPR), is a technique that expresses a certain image or pattern by assembling numerous tiny tiles. Mosaics have been used for decoration of floors and walls, from ancient times up to today, and some researchers of computer graphics have conducted various studies on the characteristics and disposition of tiles that compose a mosaic. Battiato et al. [1] defined a mosaic as an example of discrete primitive-based images and classified mosaics into multi-picture and tile mosaics depending on whether the tiles are images. A multi-picture mosaic uses a technique that reconstructs a source image by applying many images stored in a database as tiles. Photo [2, 3] and puzzle image [4, 5] mosaics belong to this category. By contrast, a tile mosaic is a mosaic reproduction as a type of traditional decorative art. Ancient tile mosaics [611] and crystallization mosaics [12, 13] are some examples of tile mosaics. In this study, we focus on tile mosaics.
The direct method, which is a common approach for creating a tile mosaic, is as follows: first, the layout or design of the mosaic is determined. An important region of the design and a guideline upon which the tiles are attached are sketched on the base plane. Prepared tiles are cut for fitting to the form of the region and are attached along a guideline. Finally, the gap between the tiles is filled with grout [14, 15].
The layout or design of a mosaic, which is first decided through a mosaic creation process, is called Andamento. Andamento is a Latin word that refers to the flow or movement of tiles and is classified according to the characteristics and positions of the tiles, as shown in Fig. 1.

Opus Tesselatum: The tiles are laid along regular horizontal lines like bricks.

Opus Classicum: Opus Tesselatum is used as the background.

Opus Vermiculatum: An outline technique for achieving a halo effect.

Opus Musivum: A repeated outline background is used.

Opus Palladianum: Tiles with an irregular shape are laid irregularly.

Opus Sectile: Complete shaped tiles are used instead of lots of individual tiles.

Fig. 1. Various types of Andamento: (a) Opus Tesselatum, (b) Opus Classicum, (c) Opus Vermiculatum, (d) Opus Musivum, (e) Opus Palladianum, and (f) Opus Sectile.

Andamento is usually shown in an ancient tile mosaic artwork and is used to determine the layout of modern mosaic art. However, most studies simulating a tile mosaic seem to focus on the tile mosaic of a certain Andamento (mainly Opus Musivum), despite not mentioning the term Andamento.
In this paper, we describe the characteristics of mosaics and Andamento and propose an energy function-based mosaic framework for creating a mosaic of various types of Andamento. We classify Andamento according to certain characteristic of the tiles and represent the energy function as the weighted sum of the energy terms used to evaluate each characteristic. The closer a mosaic evaluated based on the energy terms is to a certain Andamento, which is represented as the weighted sum of the energy terms corresponding to the tile characteristics, the lower the value of the energy function. We propose an energy-function-based optimization algorithm to minimize the energy function. Our algorithm consists of a global refinement, which is a fast approximation, and a local refinement for finding the optimized solution, and seeks the solution both gradually and iteratively.
The contribution of this paper is as follows. In this paper, we study Andamento, which is a traditional mosaic style, and demonstrate a mathematical way for representing Andamento via a mosaic energy function. Then, we propose an iterative algorithm for optimizing the energy function and show that proposed algorithm successfully generates mosaic-style images of various types of Andamento.
The rest of this paper is organized as follows. Section 2 addresses previous related studies. In Section 3, we classify Andamento according to the characteristics of the tiles and describe the energy term used to evaluate each characteristic and energy function, which is the weighted sum of the energy terms. We also present an optimization process for minimizing the energy function and a post processing for achieving an effective mosaic representation. We present our results in Section 4. Finally, we discuss the conclusions and future research in Section 5.

Related Work

The first study on creating a mosaic image was conducted by Haeberli [12]. Although the author did not use the term “mosaic,” his cellular characteristic image became the starting point of the various studies on mosaics. He laid z-buffered cones on a canvas screen to create a Voronoi diagram and made the image into a mosaic by filling each cell with the color of the sample point. His idea was also employed in Hausner’s study [6], which simulated decorative mosaics by packing directional square tiles. Hausner [6] placed square tiles on the cell of a centroidal Voronoi diagram (CVD) using Lloyd’s algorithm [16] and z-buffered cones using graphics hardware [18]. Each tile faced the perpendicular direction of the near edge, and the edge was emphasized using the edge avoidance method to avoid edges. Fritzsche et al.’s interactive mosaic design method [7] enhanced Hausner’s approach by making the direction of the tile coherent and the distribution of the tile even. Although Hausner [6] and Fritzsche et al. [7] did not mention it explicitly, their studies are similar to those of Opus Musivum.
Dobashi et al. [13] expanded on the study by Haeberli [12]. They created a mosaic using a Voronoi diagram, similar to Haeberli [12]. To minimize the difference between the source image and mosaic, they defined the energy function and optimized it by moving the Voronoi cells iteratively. Their method generated tessellation-preserving edges, resulting in an Opus Palladianum image.
Elber and Wolberg [8] proposed a mosaic method that employs offset curves. They extracted the edge, which is one of the important features of an image, and generated offset curves from it. They approximated the offset curves by employing a z-buffered Voronoi diagram and laid tiles to follow the offset curves. The tiles of their visually pleasing mosaic “flow” following the offset curves, similar to the concept of Opus Musivum.
Battiato et al. [1, 10] used the term, Andamento, in their studies, and simulated an Opus Musivum and Opus Vermiculatum (although we believe it is actually an Opus Classicum). Although they did not propose a method for making various types of Andamento other than those above, their study is the first to explicitly mention Andamento among the studies on mosaics.
In addition, Schlechtweg et al. [11] developed a multi-agent system that can perform various stroke-based rendering as well as create a mosaic. In their system, each RenderBot renders a stroke according to the stroke characteristics. If tiles are used as strokes, then a mosaic that is similar to the work of Hausner [06] or Fritzsche et al. [7] is achieved.
Hertzmann [18, 19] employed an energy function for his study on painterly rendering. He defined the energy function as the sum of four energy terms that deal with the color difference between the source and result, the area of the brush stroke, the number of brush strokes on a pixel, and the number of empty pixels. He created a painterly video sequence that maintained the temporal coherence by optimizing the energy function. Similarly, Kim and Pellacini [4] also employed an energy function for their Jigsaw Image Mosaics (JIM) that filled an image with arbitrarily shaped tiles. They proposed energy terms that measure the area of overlap, color difference, and amount of deformation, and presented an optimization algorithm that minimizes the weighted sum and energy function. In their study, a mosaic of various styles (e.g., JIM, photomosaic, and simulated tile mosaic) could be obtained by adjusting the weight of each energy term. We employed this concept to generate a mosaic of various types of Andamento.
Recently, deep neural networks (DNNs) are utilized in various domains [2024]. The DNNs are also used in image stylization field, called the deep neural style transfer, which renders a content image in different styles [2535]. This algorithm takes two images—a content image and a style image—and blends them together so the output image looks like the content image, but stylized in the style of the style image. In the Section 4, we show the comparison between our results and style transferred mosaic images.

Generating Mosaic using Andamento

Andamento and Energy Function
We classify Andamento according to several characteristics of tiles, as shown in Table 1, to create an energy function that represents the various types of Andamento.

Table 1. Classification of Andamento
Andamento Detailedness of tile Regularity of tile Degree of tile alignment Type of guideline
Opus Tesselatum Medium High High Horizontal lines
Opus Musivum Medium High High Offset outline curves
Opus Palladianum Low Low - -
Opus Sectile High Low - -
The detailedness of the tile indicates how well each tile represents a source image. We define a mosaic closer to a source image as having higher detailedness. Therefore, the detailedness of Opus Sectile, which is the method of cutting tiles exactly to fit an object of the source image is high. However, the detailedness of Opus Palladianum, which consists of irregular tiles, is low.
The regularity of a tile indicates its uniformity of shape and size. The regularities of Opus Tesselatum and Opus Musivum, which have backgrounds filled with regularly shaped and sized tiles, are high. However, the regularities of Opus Sectile and Opus Palladianum are low.
The degree of tile alignment indicates how evenly the tiles align following a guideline. In the case of Opus Musivum, as tiles are laid on a guideline ringing with an offset from the outline of the object on the source image to the background, the degree of alignment is high. The degree of alignment of Opus Tesselatum is also high because it has background tiles located along horizontal guidelines.
Because Opus Classicum can be represented by composing Opus Tesselatum and other types of Andamento, we did not classify it.
We define an energy function through the following equation, organized into three terms evaluating each tile characteristic:

$E=w_d⋅E_d+w_r⋅E_r+w_a⋅E_a (-1≤w_d,w_r,w_a≤1),$(1)

where E_d, E_r, and E_a are energy terms that evaluate the detailedness, regularity, and degree of alignment of the tiles and are minimized when the corresponding characteristics of the tiles are satisfied. In other words, the higher the detailedness, regularity, and degree of alignment of the tiles, the lower E_d, E_r, and E_a are. Therefore, if we adjust the weight of each energy term, w_d,w_r,w_a, and make a mosaic by minimizing E, we can obtain a mosaic that has each tile characteristic. If the weight of an energy term is less than zero, the corresponding energy term should be increased to minimize the energy function. Therefore, a negative weight indicates non-detailedness, non-regularity, and non-alignment.

$E_d = \frac{1}{N} \displaystyle\sum_{i=1}^{N} \Bigg| \frac{k}{n} \displaystyle\sum_{(x,y)∈t_i}‖ \overline{C_{t_i}(x,y)}-C_{t_i}(x,y)‖\Bigg|$(2)

The energy term for the detailedness of the tile, E_d, is defined above. Here, C_(t_i ) (x,y) is the color of the pixel in the source image corresponding to tile t_i, n is the number of pixels of a tile, N is the number of tiles, and ‖⋅‖ is the Euclidian distance in the RGB space. Because we regard the color of each tile as the average color of the tile region on the source image, Eq. (2) evaluates the detailedness of the tile well. We used a normalizing variable, n=2.

$E_d = \frac{1}{N} \displaystyle\sum_{i=1}^{N} \Bigg| \frac{k}{n} \displaystyle\sum_{(x,y)∈t_i} ‖ \overline{C_{t_i}(x,y)}-C_{t_i}(x,y)‖ \Bigg|$(3)

$E_{size} = \frac{1}{N} \displaystyle\sum_{t∈T} \frac{|\overline S-S_t|}{\overline S}$(4)

$s_a (i,j)=\int_0^1 \frac{(T(i)(s)-T(j)(s)+θ_0}{π})^2 ds$(5)

$E_{shape} = \frac{1}{(N(N-1)} \displaystyle\sum_{t_i∈T} \displaystyle\sum_{t_j∈T} s_a (t_i,t_j )$(6)

An energy term regarding the regularity of a tile, E_r consists of E_size related to the regularity of tile size and E_shape related to the regularity of tile shape. Here, E_size (Eq. (4)) is evaluated as the average of the difference between each tile size and the average tile size, and E_shape (Eq. (6)) evaluates the shape regularity by employing a similarity measure [36] (Eq. (5)), which estimates the similarity between the two shapes using the tangent function. By calculating the similarity between all tiles, Eq. (6) evaluates the regularity of the shapes of the tiles. By properly adjusting the weight value, we can control the sensitivity between the size and shape of the tile.

$E_a=w_{position} E_{position}+w_{edge} E_{edge} (w_{position}+w_{edge}=1)$(7)

$E_{position} = \frac{1}{N} \displaystyle\sum_{i=1,t_i∈T}^{N}((dist(t_i,t_i^r )/h)^2+(dist(t_i,t_i^l )/h)^2 )$(8)

$dif(e,p,g) = \begin{cases} e⋅g&(p⋅g>0.5)\cr 1-e⋅g&(p⋅g≤0.5) \end{cases}$(9)

$E_{edge} = \frac{1}{N} \displaystyle\sum_{i=1,t_i∈T}^{N} \left( \frac{1}{m} \displaystyle\sum_{j=1}^{m} dif(e_{t_i}^j,p_{t_i}^j g_{t_i})^2 \right)^2$(10)

The energy term for the degree of tile alignment, $E_a$, consists of two terms, $E_{position}$ and $E_{edge}$. Here,
$E_{position}$ evaluates the degree of tile-position alignment using a guideline. When an artist makes a mosaic, the artist lays tiles on a pre-drawn guideline. However, we do not have explicit guidelines that provide information on the tile position. We assume that each tile lays on a “potential” guideline. If two potential guidelines of two neighboring tiles are equivalent, the two tiles are aligned because they are on the same guideline. We extracted the streamlines from the source image (detailed in Section 4.1) and defined each potential guideline as the streamline passing the center of each tile. We evaluated how well each tile aligns by measuring the distance between two potential guidelines. Here, N is the number of tiles, and h is the average height of the tiles, which is used to normalize $E_{position}$.
Here, $E_{edge}$ evaluates the degree of tile-edge alignment using a guideline. In general, when an artist makes a mosaic such as Opus Musivum and Opus Tesselatum, the artist may align the edge of each tile with a guideline. We measure $E_{edge}$ as the difference between the edge direction and the guideline direction using e, which is a direction vector of the edge of the tile; p, which is a vector from the center of the tile to the center of the edge; and g, which is the direction vector of a guideline (Eq. (9)). We calculated the difference between the direction vector of an edge of the tile and a guideline for all edges of all tiles (Eq. (10)). Here, N is the number of tiles, and m is the number of edges of each tile, and is used for normalizing $E_{edge}$.

Optimization of Energy Function
The energy function used in this study has an extremely complex dimensionality, and thus finding an optimized solution is quite difficult. We propose an iterative approach that does not guarantee the global optima but gradually finds a better solution. Evaluating an energy function at each time until the function reaches the optima requires an extremely high cost. We apply a global refinement, which converges E roughly on the initial tiles, and makes tiles by employing a local refinement that controls the characteristics of each tile to decrease E. The pseudocode below represents our highly abstract algorithm (Fig. 2).

Fig. 2. Pseudocode of proposed algorithm.

The global refinement, which is a heuristic approach, does not guarantee that E will decrease monotonically. By contrast, the local refinement, which refines each tile after evaluating E, guarantees a monotonic decrease. However, the converged value of E is not guaranteed to be the optimal solution. Therefore, our algorithm does not generate a mosaic minimizing E. Nevertheless, the local optima generated by our algorithm are sufficiently visually pleasing and contribute to enriching the results.

Pre-processing
The foreground and background of Opus Classicum and Opus Musivum have different appearances. In addition, Opus Palladianum is commonly used in the background. Therefore, our algorithm should be able to divide a source image into foreground and background images. Although there have been many studies on background extraction, which is one of the most prominent research topics in computer vision, dealing with this in detail is beyond the scope of this study. We manually selected and merged the background region using mean-shift image segmentation [37].
In most mosaics other than Opus Sectile and Opus Palladianum, each tile has an individual direction, and thus the direction of the tile is one of its important features. To obtain a spatially coherent direction, we employ the edge tangent field (ETF) [38] calculated from a source image. We made the ETF and set the direction of the tile perpendicular to the flow direction at the center of each tile. When the user defines the guideline for the background region, we obtain the flow direction of the background region from the guideline. By doing so, the user is able to select between the horizontal line and an outline that divides the foreground and background as a guideline for the background region. We then obtain the flow field by diffusing the direction of the guideline toward the background region.

Global tile refinement
In this section, we estimate the rough position of the tiles. Because we do not make any other tiles yet, accurately calculating E is impossible. We propose a heuristic and an iterative approach to finding the initial position of the tile that is estimated to rapidly lower $E$.
First, we randomly generate seed points that correspond to the center of each tile. At each seed point, we find the vectors $v_d$, $v_r$, and $v_a$, which are toward the position estimated to reduce $E_d$, $E_r$, and $E_a$. We assume that the position indicated by the weighted sum of these vectors, $v_E=w_d⋅v_d+w_r⋅v_r+w_a⋅v_a$ reduces $E$ rather than the current position and moves each seed point along $v_E$. If $w_d$, $w_r$, and $w_a$ are negative, $v_d, v_r$, and $v_a$ are in the opposite direction. Therefore, the seed points are moved toward the direction, increasing $E_d, E_r$, and $E_a$. These processes are repeated until the position of the entire seed point converges or $|v_E|$ becomes less than the pre-defined threshold value.
We present the method for finding $v_d, v_r$, and $v_a$ in the following.

Estimating detailedness energy term
We assume that the value of $E_d$ is small if the tiles are located in a homogeneous colored region of the source image. A homogeneous colored region indicates that the density of a certain color is extremely high. We obtain the mean shift vector toward the position where the colors similar to the color of the seed point are densely distributed by employing a kernel density estimator [37]. The following equation is a kernel density estimator using the d-variate kernel K, which has an h-sized window.

$\hat f(x) = \frac{1}{(nh^d} \displaystyle\sum_{i=1}^{n} K \left( \frac{x-x_i}{h}\right)$(11)

We use $\sqrt{n/WH}$ as the value of $h$. Here, $W$ and $H$ are the width and height of the source image, respectively, and $n$ is the number of seed points.

Estimating regularity energy term
We intuitively know that $E_r$ generally decreases if the seed points that are tiles are evenly distributed. By employing a particle repulsion technique, we define the repulsion direction of each tile as $v_r$. We modified the method of Witkin and Heckbert [39] as follows:

$w_i=\begin{cases} (kr_θ - |v_i|)^2 / |v_i| & (0 < |v_i| < kr_θ) \cr 0 & (otherwise) \end{cases}$(12)

$v_r=\displaystyle\sum_{i}^{n}-w_i v_i$(13)

Here, $v_i$ is a vector toward another seed point i from the current seed point, and $r$ is the radius of the repulsion kernel. Because we do not limit the shape of the tiles to a square, we apply repulsion to the non-square tile using an elliptic kernel. We define r as the radius of the ellipse whose area is WH/n and whose semimajor-to-semiminor ratio is given by the user. We rotated the elliptic kernel to make the semi-major axis face the flow direction at the center of the kernel. We used this kernel after k-times scaling. A value of k=3 works well.

Estimating alignment energy term
Because we do not generate any tiles yet, it is difficult to estimate $E_{edge}$. However, E_position can be estimated because the seed points will be the location of the tiles. We obtain the potential guideline for each seed point and calculate $v_a$ for reducing $dist(t_i,t_i^r)^2+dist(t_i,t_i^l )^2$ (in Eq. (8)). To obtain the neighboring seed points of a certain seed point, we apply the Voronoi tessellation temporally. We find the Voronoi edge, which is intersected with a potential guideline passing on the seed point, and select seed points of Voronoi cells that share that edge. We obtain two vectors $v_a^1$ and $v_a^2$ at two seed points toward a potential guideline and calculate the sum of the two vectors as $v_a$.

Local refinement of tiles using edge transforms
Now, while evaluating $E$, we manipulate each tile to minimize $E$. Before this, we create the initial tile from the Voronoi cell at each seed point. In this section, we present an edge transform-based optimization for minimizing $E$. Edge transform-based optimization finds candidates for an edge transform, which is estimated to reduce E, and after evaluating $E$, applies the best candidate transform only if $E$ is reduced. We propose a method for finding the transform candidates that correspond to the five terms of $E$. We find five transform candidates for the edge of the tile, and if the weight is negative, we conduct a transformation in the opposite direction. This process is applied for all edges of the tiles and repeated until E is converged. The center of each tile is recalculated at all iterations, and it is assumed that the color of the tile is the average color of the tile region.

Finding an edge transform for detailedness
We propose a method for finding a candidate for an edge transform that is expected to reduce $E_d$. We use a joint kernel density estimator whose kernel consists of two half ellipses. Each half-ellipse kernel is located at each tile on both sides of an edge and is joined while sharing the edge.

$\hat f(x) = \frac{1}{(n+m)h^d}\left(\displaystyle\sum_{i=1}^{n} K \left(\frac{c_1-x_i}{h}\right) + \displaystyle\sum_{i=1}^{m} K \left(\frac{c_2-x_i}{h} \right) \right)$(14)

The semi-major radius of each half-ellipse kernel is half of the edge length, and the semi-minor radius of each half ellipse kernel is the smaller of two distances between the edge and the center of both tiles (c_1 and c_2). Each kernel has a center point whose color is the average color of the opposite tile. This kernel density estimator returns a mean shift vector that indicates the proper location of the edge. We elect an edge transform candidate that consists of a translation along the mean shift vector direction and rotation to make the edge perpendicular to the mean shift vector direction.

Finding an edge transform for size regularity
We elect an edge translation transform candidate expected to reduce E_size. The magnitude of the translation vector is defined as follows:

$mag = \begin{cases} \frac{|\overline s-s_1| + |\overline s-s_2|}{2l} & ((\overline s-s_1 )(\overline s-s_2 ) < 0)\cr 0&(otherwise) \end{cases}$(15)

Here, $s_1$ and $s_2$ are the sizes of both tiles sharing the edge, $\overline s$ is the average size of all tiles, and l is the length of the edge. We obtain the vector whose magnitude is above and direction is toward the tile whose size is more different than the average size of the tile. The edge transform candidate consists of a translation that moves an edge toward the spot indicated by that vector.

Finding an edge transform for shape regularity
We find an edge rotation transform candidate that is anticipated to decrease $E_{shape}$. An edge rotation transform rotates an edge around the center of the edge. The equation for determining the rotation angle is given below:

$θ_j (a,b) = \frac{1}{n-1} \displaystyle\sum_{i=1}^{n} \frac{\int_a^b(T(i)(s)-T(j)(s)+θ_0 ) ds}{b-a} (i≠j)$(16)

Here, a and b indicate the start and end points of the current edge when the circumference of the tile is normalized to 1, and T is a tangent function of Eq. (5). This equation leads to finding a vector that indicates a location that decreases the sum of the difference of the tangent function. This edge rotation transform, which rotates an edge along the above angle, is selected as a candidate for the edge transform.

Finding an edge transform for tile alignment
To align the tiles, we use $v_a$ calculated in Section 4.2. In Section 4.2, we move the seed points corresponding to the center of tiles, whereas in this section we move the edge instead of the center of the tile. If the edge is moved, the center of the tile is recalculated. Therefore, the center of the tile moves toward the direction of the edge movement. We make an edge translation transform candidate to move the edge along the sum vector of $v_a$ of both tile shared edges.

Finding an edge transform for tile edge alignment
We obtain the equation below by modifying Eq. (9), which finds $E_{edge}$.

$θ_j (b,p,g)=\begin{cases} arccos⁡(b⋅g)&(p⋅g>0.5)\cr arccos⁡(1-b⋅g)&(p⋅g≤0.5) \end{cases}$(17)

The above equation rotates an edge perpendicular or parallel to the guideline after considering the relative position (indicated by vector p) of the edge in the tile. An edge rotation of the above angle is selected as a candidate for an edge transform.

Post-processing
To obtain an aesthetically pleasing result, we conducted some post-processing on the result of Section 4.3. To add a more vivid expression of tiles, we mapped a texture obtained from a real mosaic to our tiles. We use a method in which the user selects a texture manually or a method that retrieves the proper texture from the database automatically. With the automatic method, we obtained the textureness map of the source image and the texture by employing the method in [40], and evaluated the structure similarity (SSIM) [41] between the textureness map of the source image and texture. We selected the texture with the best SSIM. When we map the texture, we only use the intensity of the texture. Hue is obtained from the source image.
There is no grout in the mosaic of Section 4.3. However, grout is a very noticeable feature of a mosaic for both aesthetic and practical reasons. To generate a grout region, we jitter the vertices toward the inside of the tile.
We used a texture synthesis for an effective expression of the grout. We synthesized a grout texture similar to the method used in stained glass rendering [42], which obtains a boundary texture by employing the image analogy method [43]. As a grout image of a real mosaic image (A'), the grout mask of the real mosaic (A), and the grout mask of our mosaic (B) have the following relationship, and we can synthesize a grout image (B').

$A:A′: :B:B′$(18)

Results

Fig. 3(a) shows the results for Opus Mussivum using our algorithms. Here, $w_d$ is 0.5, $w_r$ is 0.8, and $w_a$ is 0.8. We can note here that the extended outline, which is a characteristic of Opus Mussivum, is formed properly by the distance matrix. It can be seen that Opus Vermicullatum is applied even to the rectangular boundary of the inputted raster image, which is different from real Opus Mussivum.
Fig. 3(b) shows the output image to which Opus Classicum was applied. In the background regions, $w_d$ is 0.0, $w_r$ is 0.8, and $w_a$ is 0.8. In the foreground region, the values used for generating Opus Mussivum are used. It can be seen that the tesserae are arranged horizontally forming lines in the background. In this way, we have the effect of emphasizing the object of the foreground.
Fig. 3(c) shows the results for Opus Palladianum. Here, $w_d$ is 0.0, $w_r$ is 0.2, and $w_a$ is 0.0. Perturbations of the tesserae scale were applied.

Fig. 3. Mosaic results of different types of Andamento: (a) Opus Mussivum, (b) Opus Classicum, and (c) Opus Palladianum.

Fig. 4. Various results of proposed mosaic generation methods: (a) Opus Mussivum, (b) Opus Classicum, and (c) OpusMussivum.

Fig. 4 shows various mosaic results produced through the proposed method.
Neural style transfer algorithms can produce images with various styles while preserving content. This concept is similar to our goal which stylizes mosaic images with various style, Andamento. Fig. 5 shows some style-transferred mosaic images. For these style transfers, the MobileNet [35] and InceptionV3 [36] models are used. As shown in Fig. 5, deep neural style transfer algorithms are not suitable for transferring ancient tile patterns to input content image, which is Lenna image, here. Consequently, our method is significantly useful for stylizing mosaics.
Fig. 5. Results of style-transferred mosaic images.

Conclusion

In this paper, we proposed an energy-function-based mosaicing framework for expressing various types of Andamento. We classified Andamento according to the three characteristics of the tile, the detailedness of the tile, the regularity of the tile, and the degree of tile alignment. We organized the energy terms to evaluate each characteristic. We also proposed an optimization algorithm that minimizes an energy function that consists of three terms. Our method effectively generates several types of Andamento using our single framework and creates a visually pleasing mosaic image using various textures and grout synthesis.
We implemented an efficient tile refinement by employing a Voronoi diagram. However, a curved edge of a tile seen on Opus Sectile in a real mosaic cannot be generated by our Voronoi diagram-based method. Therefore, a study on a structure for making a curved edge is needed. It is also expected that the mean shift vector calculation will be accelerated drastically by employing parallel computing of graphics hardware. Therefore, acceleration using a GPU, the computing cost of which is extremely high, is one of the most important future areas of this study. In addition, our framework, which controls the style of a mosaic, i.e., Andamento, can be employed in other studies on stroke-based rendering. If various strokes, such as points, lines, and brushes, instead of tiles, are used in our framework, stroke-based paint images with various styles are expected to be obtained.

Acknowledgements

Not applicable.

Author’s Contributions

Conceptualization, DK. Funding acquisition, DK. Investigation and methodology, SS. Project administration, SS. Resources, DK. Supervision, DK. Writing of the original draft, DK. Writing of the review and editing, DK, SS. Software, DK. Validation, DK, SS. Formal analysis, DK, SS. Data curation, DK, SS. Visualization, DK. All the authors have proofread the final version.

Funding

This study was supported by the Research Program funded by the Seoul National University of Science and Technology (SeoulTech).

Competing Interests

The authors declare that they have no competing interests.

Author Biography

Name : Dongwann Kang
Affiliation : Seoul National University of Science and Technology
Biography : Dongwann Kang is an Assistant Professor in the Department of Computer Science and Engineering at Seoul National University of Science and Technology. He received his B.S. and Ph.D. degrees from Chung-Ang University in Korea in 2006 and 2013, respectively. In Korea, he was a lecturer at Seoul National University and Sookmyung Women's University. In the UK, he was a visiting researcher and a Marie Sklodowska-Curie fellow at the Faculty of Science and Technology, Bournemouth University. His research interests include non-photorealistic rendering and animation, digital imaging, affective computing, and GPU processing.

Name : Sanghyun Seo
Affiliation : Chung-Ang University
Biography : Sanghyun Seo received his B.S. degrees in Computer Science and Engineering from Chung-Ang University, Seoul, Korea, in 1998 and M.S. and Ph.D. degrees in GSAIM Dep. at Chung-Ang University, Seoul, Korea, in 2000 and 2010. He was a postdoctoral researcher at LIRIS Lab, Lyon 1 University from February 2011 to February 2013. He had worked at the ETRI, Daejeon, Korea, May 2013 to February 2016 and at the Department of Media Software at Sungkyul University, Anyang-si, Korea, March 2016 to February 2019. Now, he is currently an associate professor in the College of Art & Science, Chung-Ang University.

References

[1] S. Battiato, G. Di Blasi, G. M. Farinella, and G. Gallo, “Digital mosaic frameworks: an overview,” Computer Graphics Forum, vol. 26, no. 4, pp. 794-812, 2007.
[2] R. Silvers and M. Hawley, Photomosaics. New York, NY: Henry Holt and Company, 1997.
[3] A. Finkelstein and M. Range, “Image mosaics,” in Electronic Publishing, Artistic Imaging, and Digital Typography. Heidelberg, Germany: Springer, 1998, pp. 11-22.
[4] J. Kim and F. Pellacini, “Jigsaw image mosaics,” ACM Transactions on Graphics, vol. 21, no. 3, pp. 657-664, 2002.
[5] J. Orchard and C. S. Kaplan, “Cut-out image mosaics,” in Proceedings of the 6th International Symposium on Non-photorealistic Animation and Rendering, Annecy, France, 2008, pp. 79-87.
[6] A. Hausner, “Simulating decorative mosaics,” in Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 2001, pp. 573-580.
[7] L. P. Fritzsche, H. Hellwig, S. Hiller, and O. Deussen, “Interactive design of authentic looking mosaics using Voronoi structures,” in Proceedings of International Symposium on Voronoi Diagrams in Science and Engineering (VD), Seoul, Korea, 2005.
[8] G. Elber and G. Wolberg, “Rendering traditional mosaics,” The Visual Computer, vol. 19, no. 1, pp. 67-78, 2003.
[9] G. Di Blasi and G. Gallo, “Artificial mosaics,” The Visual Computer, vol. 21, no. 6, pp. 373-383, 2005.
[10] S. Battiato, G. Di Blasi, G. M. Farinella, and G. Gallo, “A novel technique for opus vermiculatum mosaic rendering,” in Proceedings of the 14th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, Plzen, Czech Republic, 2006, pp. 133-140.
[11] S. Schlechtweg, T. Germer, and T. Strothotte, “RenderBots: multi-agent systems for direct image generation,” Computer Graphics Forum, vol. 24, no. 2, pp. 137-148, 2005
[12] P. Haeberli, “Paint by numbers: abstract image representations,” in Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, Dallas, TX, 1990, pp. 207-214.
[13] Y. Dobashi, T. Haga, H. Johan, and T. Nishita, “A method for creating mosaic images using Voronoi diagrams,” in Proceedings of the 23rd Annual Conference of the European Association for Computer Graphics (Eurographics) (Short Presentations), Saarbrucken, Germany, 2002.
[14] S. Kelly, J. Docherty, A. Read, and R. Wates, Complete Mosaic Handbook: Projects, Techniques, Designs. Richmond Hill, Canada: Firefly Books, 2004.
[15] K. Fassett and C. Bahouth, Mosaics: Inspiration and Original Projects for Interiors and Exteriors. Newtown, CT: Taunton Press, 2001.
[16] S. Lloyd, “Least squares quantization in PCM,” IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 129-137, 1982.
[17] K. E. Hoff III, J. Keyser, M. Lin, D. Manocha, and T. Culver, “Fast computation of generalized Voronoi diagrams using graphics hardware,” in Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 1999, pp. 277-286.
[18] A. Hertzmann, “Painterly rendering with curved brush strokes of multiple sizes,” in Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, Orlando, FL, 1998, pp. 453-460.
[19] A. Hertzmann, “Paint by relaxation,” in Proceedings of Computer Graphics International, Hong Kong, China, 2001, pp. 47-54.
[20] Q. Ren, “A video expression recognition method based on multi-mode convolution neural network and multiplicative feature fusion,” Journal of Information Processing Systems, vol. 17, no. 3, pp. 556-570, 2021.
[21] Y. Zhou, “Vehicle image recognition using deep convolution neural network and compressed dictionary learning,” Journal of Information Processing Systems, vol. 17, no. 2, pp. 411-425, 2021.
[22] U. Jang, K. H. Suh, and E. C. Lee, “Low-quality banknote serial number recognition based on deep neural network,” Journal of Information Processing Systems, vol. 16, no. 1, pp. 224-237, 2020.
[23] C. Ren, D. K. Kim, and D. Jeong, “A survey of deep learning in agriculture: techniques and their applications,” Journal of Information Processing Systems, vol. 16, no. 5, pp. 1015-1033, 2020.
[24] S. Bibi, A. Abbasi, I. U. Haq, S. W. Baik, and A. Ullah, “Digital image forgery detection using deep autoencoder and CNN features,” Human-centric Computing and Information Sciences, vol. 11, article no. 32, 2021. https://doi.org/10.22967/HCIS.2021.11.032
[25] Y. Xu, Y. Li, and B. S. Shin, “Medical image processing with contextual style transfer,” Human-centric Computing and Information Sciences, vol. 10, article no. 46, 2020. https://doi.org/10.1186/s13673-020-00251-9
[26] Y. Jing, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, “Neural style transfer: a review,” IEEE Transactions on Visualization and Computer Graphics, vol. 26, no. 11, pp. 3365-3385, 2020.
[27] N. Q. Tuyen, S. T. Nguyen, T. J. Choi, and V. Q. Dinh, “Deep correlation multimodal neural style transfer,” IEEE Access, vol. 9, pp. 141329-141338, 2021.
[28] H. Wang, Y. Li, Y. Wang, H. Hu, and M. H. Yang, “Collaborative distillation for ultra-resolution universal style transfer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, 2020, pp. 1857-1866.
[29] P. Wang, Y. Li, and N. Vasconcelos, “Rethinking and improving the robustness of image style transfer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual Event, 2021, pp. 124-133.
[30] D. Kotovenko, M. Wright, A. Heimbrecht, and B. Ommer, “Rethinking style transfer: from pixels to parameterized brushstrokes,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Virtual Event, 2021, pp. 12196-12205.
[31] S. Liu, T. Lin, D. He, F. Li, M. Wang, X. Li, Z. Sun, Q. Li, and E. Ding, “AdaAttN: revisit attention mechanism in arbitrary neural style transfer,” in Proceedings of the IEEE International Conference on Computer Vision, Virtual Event, 2021, pp. 6649-6658.
[32] L. Du, “How much deep learning does neural style transfer really need? An ablation study,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Snowmass Village, CO, 2020, pp. 3150-3159.
[33] Z. Li, F. Zhou, L. Yang, X. Li, and J. Li, “Accelerate neural style transfer with super-resolution,” Multimedia Tools and Applications, vol. 79, no. 7, pp. 4347-4364, 2020.
[34] J. J. Virtusio, J. J. M. Ople, D. S. Tan, M. Tanveer, N. Kumar, and K. L. Hua, “Neural style palette: a multimodal and interactive style transfer from a single style image,” IEEE Transactions on Multimedia, vol. 23, pp. 2245-2258, 2021.
[35] X. Wang, Y. Lyu, J. Huang, Z. Wang, and J. Qin, “Interactive artistic multi-style transfer,” International Journal of Computational Intelligence Systems, vol. 14, no. 1, pp. 1-13, 2021.
[36] E. M. Arkin, L. P. Chew, D. P. Huttenlocher, K. Kedem, and J. S. Mitchell, “An efficiently computable metric for comparing polygonal shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 3, pp. 209-216, 1991.
[37] D. Comaniciu and P. Meer, “Mean shift: a robust approach toward feature space analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, 2002.
[38] H. Kang, S. Lee, and C. K. Chui, “Coherent line drawing,” in Proceedings of the 6th International Symposium on Non-photorealistic Animation and Rendering, San Diego, CA, 2007, pp. 43-50.
[39] A. P. Witkin and P. S. Heckbert, “Using particles to sample and control implicit surfaces,” in Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, Orlando, FL, 1994, pp. 269-277.
[40] S. Bae, S. Paris, and E. Durand, “Two-scale tone management for photographic look,” ACM Transactions on Graphics, vol. 25, no. 3, pp. 637-645, 2006.
[41] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.
[42] S. Brooks, “Image-based stained glass,” IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 6, pp. 1547-1558, 2006.
[43] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, “Image analogies,” in Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 2001, pp. 327-340.

Dongwann Kang1 and Sanghyun Seo2,*, Mosaic Stylization Using Andamento, Article number: 12:09 (2022) Cite this article 2 Accesses