Example-Based Hairstyle Advisor

Example-Based Hairstyle Advisor Wei Yang, Masahiro Toyoura and Xiaoyang Mao University of Yamanashi,Japan Abstract Hairstyle is one of the most important features to characterize one s appearance. Whether a hairstyle is suitable or not is said to be closely related to one s face shape. This paper proposes a new technique for automatically finding one s suitable hairstyle from a collection of successful hairstyle examples. A hair-face image composition method utilizing modern matting techniques is also developed for the synthesis of realistic hairstyle images. The effectiveness of the proposed technique has been validated through a subject study. 1. Introduction Hairstyle is one of the most important features to determine one s appearance and mood. A person can look completely different simply by changing her hairstyle. Everyone would like to have a suitable hairstyle making her look attractive, but it is usually difficult to find such one as we cannot easily try out various styles with our real hair. Several commercial or free software have been developed allowing users to simulate their look in different hairstyles by manually selecting hairstyle samples and superimposing them over their face images[1-2]. Although those systems do provide some general guideline on hairstyle choosing, they do not provide any hints on what s the suitable hairstyle for a particular face. Therefore the user usually needs to go through a very tedious process trying out many different hairstyles before the desired one can be found. On the other hand, in the field of computer graphic, there exist several excellent works related with hairstyle[3] or [4] or [5] or [6]. However, to the best of our knowledge, all those works focused on how to model and render hairstyles with computer graphics and are mainly applied for creating virtual characters and animations. In this paper, we propose a new technique for automatically finding the suitable hairstyle for a given face. There are many factors may affect how one would look to be in a particular hairstyle. We started our project by interviewing several successful hair stylists and an important fact we observed is that although there are no stylists can tell any explicit rules about their design, they all view the shape of the face as the most important attribute in desining a hairstyle. This inspired us to adopt an example-based frame work, an approach which has been successfully used for texture synthesis[7,8] and style transferring [9,10] in recent years. Our new technique finds the suitable hairstyles for a given face by learning the relationship between face shapes and the examples of successful hairstyles. The major contribution of this paper can be summarized as follows: 1. A new framework for finding suitable hairstyles from successful hairstyle examples. 2. A hair-face image composition method utilizing modern matting techniques for automatic realistic hairstyle image synthesis. 3. A subject study demonstrating the validity of the feature vector and the effectiveness of the examplebased approach. The remainder of the paper is organized as follows: After presenting the example-based framework in Section 2, we describes the design and computing of the feature vector used for search the best hairstyles in Section 3. Section 4 addresses the synthesis of realistic hairstyle image. Section 5 discusses the implementation and shows the results of evaluation experiments. Section 6 concludes the paper and shows several future research directions. 2. Example-based framework Given a face image I input, we want to create another image I output with a hairstyle S matching the face best. We realize it in two steps: Searching: Find the most similar face shape in feature vectors space and adopt the hairstyle as the suitble hairstyle S. Composition: Superimpose the hairstyle S over the face image I input to obtain a realistic image I output of the face in a suitable hairstyle.

Figure 1: System Framework 2.1 System overview As shown in Figure 1, our system assumes n successful hairstyle images I i (i=1,2 n) is available. At seraching phase, the following operations are executed to build a database set T(V i, i)(i=1,...n) where V i is the feature vector characterizing the shape of each face I i and i is the α-matte indicating the probability of hair area in the image: 1. Apply robust matting technique[11] to I i (i=1,2 n) to create i(i=1,2 n). 2. A trained ASM model is used to detect the facial feature points on I i (i=1,2 n). 3. Construct the feature vector V i (i=1,2 n) from the ASM feature points. While the first operation requires the manual specification of example strokes to create the tri-map for estimating the α-matte, other two operations are performed in a fully automatic way. At composition phase, given a face image I input,the system perform the following operations to compute a set of suitable hairstyles for I input. 1. Apply a trained ASM model to detect the facial feature points of I input. 2. Construct the feature vector V Input characterizing the shape of the face in I input. 3. Search through all the images in database in the feature vector space. 4. Sort d(v i, V input ) 5. Take the top k image I i (j=1,..k) and composite them with I input using i (j=1,..k). The design of feature vector V is crucial for obtaining a good result, as well as for a fast searching through the training examples. We will discuss it in detail in the next Section. oval round triangle home base (a)shapes of face (b) Hairstyle for round face Figure 2: Face shape and its relationship to hairstyle 3. Feature vector design ASM[12] is one of the most popular techniques for detecting the geometric features of faces from images. Instead of using ASM directly, we want to have a more compact feature vector, which can well model the relationship between face shape and hairstyles. As shown in Figure 2(a), it is known that human faces can be roughly classified into 4 categories by shape: oval, round, triangle and home base. A hairstyle giving an impression of a oval shaped face is likely to be a suitable one[13]. For example, as shown in Figure 2(b), in case of a round face, the styles with long bangs flowing smoothly toward two sides or back makes the face look longer and hence are suitable, while the one with flat cut bangs and a thick volume on top of the head is not suitable because it further emphasizing the impression of round shape. To learn such kind of relationship between face shape and hairstyle, we first compute the following 6 line segments (Figure 3(b) ):

h : the center vertical line segment w 1 : the horizontal line segment at the height of eyebrows w 2 : the horizontal line segment at the widest position of the face area w 3 : the horizontal line segment at the height of mouth h t : the vertical line segment from the top of the face to the cross-section of h and w 1 h b : the vertical line segment from the bottom of the face to the cross-section of h and w 3 from the above 6 line segments, we define a 6-dimensional feature vector V(v 1,v 2,v 3,v 4,v 5,v 6 ) with v j (j=1,2,..6) being the ratio of two line segments, normalized to be in (0,1] (Figure 3(c)). To automatically compute the feature vector, we trained the ASM model so that it includes the end points of the 6 line segments as the feature vector(figure 3(a)). The distance function is defined as d(i,i input )= (4) The coefficient k j (j=1,..6) controls the weight of each dimension in deciding the hairstyle. For example, if we use a large k 1 and a small k 2, then the system would treat the aspect ratio of the face more important and the shape of cheek less important in deciding the suitable hairstyle. Two faces with different shapes of cheek but similar aspect ratio may be suggested with similar hairstyles. In our current implementation, we set all the 6 coefficients to be equal to treat all dimensions homogenously. We plan to explore the best coefficients through machine learning in our future work. v 4 = v 5 = v 6 = (b) component of feature vector Figure 3: features vetors 4. Hairstyle image synthesis Before we can superimpose an obtained hairstyle image S over the input face image I input, position alignment and size adjustment between two images are required. This is realized by a scaling of H with / in width, / h H in height, followed by a translation aligning the upper end point of h H with that of. As shown in Figure 4(a), due to the fitting error of ASM model, we may fail to obtain a centered h H or, and this may result in a unnatural composition like the one in Figure 4(b). To correct the error, we translate h in horizontal direction by a displacement D (Figure 4 (c)) : Figure 4(d) shows an improved result obtained by using the new position of h H and for position alignment. (5) (a) error of h (b) result with the displaced h (a) result by ASM (b) feature lines v 1 = v 2 = v 3 = (c)correction of h (d)result with the corrected h Figure 4: position alignment

Finally the α-matte of S is used to composite S and I input to obtain the output image I output Here p(x,y) is the pixel, and, H p,i p, are the pixal value of the out put hairtyle image, the input image and the suitable haistyles suggested by the system, respective. Figure 5 compares the results of binary mask based composition and, our matting based method. We can see our method produces more realistic images, especially at the regions near the boundary of hair. (6) (a) with binary mask (b) -matte Figure 5: Compare the composition result using a binary mask and -matte 5. Implementation and evaluation 5.1 Implementation and result. In our current implementation, we built a training data set consisting 84 hairstyle images collected with the courtesy of hair stylists. The training data sets are divided into 3 subsets- long, medium and short, according to the length of the hairstyles. Through a preliminary user study, we found that in many cases a users want to specify the length of the hair before searching the best hairstyles, and hence it would be helpful if we can advise them the best candidates in different lengths. Currently both the hairstyle examples and the input face need to be frontal photos and the face area in the input image shouldn t be covered by hairs. In addition to the 6 dimensions representing the shape of a face, 3 additional dimensions, representing the length, hardness and volume of hair, respectively, are also added and a 9-dimensional feature vector is computed for each example. Each hair can take one of 3 values in the 3 newly added dimension: Length Volume Hardness : long, medum, short : large, medum, small : hard, medium, soft The necessity to consider the property(hardness, volume) of hair arised from the fact that the hairstyle one can actually have is constrained by the property of his/her hair, even one can find the best hairstyle making him/her look the attractive virtually. Therefore, with the additional dimensions characterizing the property of the hair, we can constrain the sampling only from the hairstyles of similar hair properties. The hair property of the input face is specified by the user. Figure 7 shows two examples of results. For the input face image at the left, the nearest face in the feature vector space for each of the long, medium and short hairstyle training set are shown in the upper row and below it is the resuling hairstyles. 5.2 Evaluation We have conducted two experiments to validate the effectiveness of our approach. The first experiment aims to investigate how the hairstyles advised by our system for a person would look to be for other people, while the second one tests how satisfactory the result is for the person himself/herself. 10 female college students participated the first experiment. We prepared 9 sets of hairstyle images, each consisting of 10 hairstyles with 2 of them were advised by the system as the suitable hairstyle. At each trial, a subject was presented with one set of images and asked to mark top 3 most suitable hairstyles out of the 10 hairstyles. Figure 6 shows an example of the hairstyle image set used in experiment. The total number of trials is 90(9 sets 10 persons) and we evaluated the probability of the occurence of the following 3 cases respectively. Case 1 Case 2 Case 3 : A system advised hairstyle were marked as the top suitable hairstyles : At least one system advised hairstyles were included in the top 2 suitable hairstyles. : At least one system advised hairstyles were included in the top 3 suitable hairstyles. Table 1: Result of Experiment 1 : Viewed by Others (binomial test) Number of occurrence Probability upon null (out of 90 trials) hypothesis Case 1 20 0.25 Case 2 58 0.00 Case 3 80 0.00 We used binomial test and our null hypothesis is that all the 10 hairstyles in an image set would be marked with the same probability. Table 1 shows the experiment results. Out of the 90 trials, there are 20 trials for case 1, 58 trials for case 2 and 80 trials for case 3, respectively. The probability for a number of occurrence above those observed ones upon the null hypothesis is shown at the rightmost column of Table 1. The null hypothesis was rejected at a significant level lower than 0.0% for the latter two cases. In other words, the experiment results suggest that the hairstyles advised by our system at least can be viewed as the second best hairstyle though might not be the best.

Table 2: Result of Experiment 2 : Viewed by Oneself (binomial test) group Number of occurrence (out of 90 trials) Probability upon null hypothesis Case 1 5 0.57 Case 2 16 0.02 Case 3 25 0.00 The second experiment had the same setting as that of the first one except for that each subject was presented with the hairstyle images of herself. The subjects are 4 female college students and 3 image sets were prepared for each of them. Therefore the total number of trials is 12(3 sets 4 persons). The number of trials for the 3 cases and the corresponding probability upon null hypothesis are shown in Table2. Same as the result of the first experiment, we can see the null hypothesis was reject with a very low level of significance for the latter two cases. From the experiment results, we can conclude that though may not be the best one, our system can advise the users the good candidates of hairstyles viewed to be suitable both by oneself and others. Fiugre 6: Example of the image set used for experiment 6. Concluding remarks We have presented a new example-based framework for creating suitable hairstyles for a given face image. Suitability is a perceptual attribute and our evaluation experiment demonstrated the effectiveness of addressing such kind of problem with an example-based approach. One big advantage of the example-based approach is that we can apply the system for a wide range of applications simply by constructing an appropriate training dataset. For example, if we use a set of hairstyles which have given a good impression to interviewer as the training examples, then our system will be very useful for a job-hunter. As one of the future research work, we want to apply the same framework to other fashion simulation problem, such as advising a suitable dressing by learning the relationships between the body shape and successful dressings. In current application a compact feature vector was designed to facilitate the fast learning of the relationship between the face shape and the successful hairstyles. There might be some other factors may affect the design of hairstyles. We are now working on finding a better feature vector by using some more sofisticated machine learning technique. References 1 Vritual Hairstudio: http://en.virtualhairstudio.com/ 2 Beauty Wizard: http://www.visoft.com/beauty/ 3 XU, Z., AND YANG, X. D. 2001. V- hairstudio: An interactive tool for hair design. IEEE Computer Graphics and Applications 21, 3, 36 43. 4 YU, Y. 2001. Modeling realistic virtual hairstyles. In PG 01: Proc. of the 9th Pacific Conference on Comp. Graphics and Applications, IEEE Computer Society,Washington, DC, USA, 295. 5 Paris, S., Chang, W., Jarosz, W., Kozhushnyan, O., Matusik, W., Zwicker, W., and Durand. F. 2008. Hair Photobooth: Geometric and Photometric Acquisition of Real Hairstyles. In ACM Transactions on Graphics (Proc. SIGGRAPH), 27(3). 6 WANG, L., YU, Y., ZHOU, K., AND GUO, B. 2009. Example based hair geometry synthesis. ACM Transactions on Graphics (Proc. of SIGGRAPH 2009) 28, 3, Article 56. 7 A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In the Seventh International Conference on Computer Version, pages 20 27, 1999. 8 A. Hertzmann, C. E. Hacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In SIGGRAPH 2001, 2001. 9 H. Chen, L. Liang, Y. Q. Xu, H. Y. Shum, and N. N. Zheng. Example-based automatic portraiture. In ACCV02, 2002 10 C.Chang, Y.Peng, Y.Chen, and S.Wang, "Artistic Painting Style Transformation Using Example-based Sampling Method,", Journal of information science and engineering, vol. 26, no.4, pp. 1443-1458, 2010. 11 J. Wang, M.F. Cohen, Optimized Color Sampling for Robust Matting, Proc. CVPR 07, pp. 1-8, 2007. 12 T.F. Cootes and C.J. Taylor and D.H. Cooper and J. Graham "Active shape models - their training and application". Computer Vision and Image Understanding (61): 38 59, 1995. 13 Best Hairsyle by Face Shape, Editted by SHUFUNOTOMO Co.,Ltd, 2007(In Japanese).

Figure 7: Example of results