2013/2/12 HEADACHED QUESTIONS FOR FEMALE. Hi, Magic Closet, Tell me what to wear MAGIC CLOSET: CLOTHING SUGGESTION

Transcription

1 HEADACHED QUESTIONS FOR FEMALE Hi, Magic Closet, Tell me what to wear Si LIU 1, Jiashi FENG 1, Zheng SONG 1, Tianzhu ZHANG 3, Changsheng XU 2, Hanqing LU 2, Shuicheng YAN 1 1 National University of Singapore 2 Institute of Automation, Chinese Academy of Science 3 Advanced Digital Science Center Search Suggestion Pairing Si Liu, et.al. Street-to-shop: Cross-Scenario Clothing Retrieval via Parts Alignment and Auxiliary Set, CVPR 2012 WHAT TO WEAR MAGIC CLOSET: CLOTHING SUGGESTION I am going on a conference, what should I wear? I am for a conference, recommend a clothing from my photo album? Attribute vs. Occasion Model Attribute vs. Attribute Model (a) suit (b) upper + lower + + Social photos of myself or friends Human Parts Alignment Clothing Attribute Estimation White Sleeveless Strapless Clothing Recommendation Model Clothing Suggestion Results MAGIC CLOSET OCCASION ORIENTED CLOTHING RECOMMENDATION STYSTEM HOW TO PAIR? Recommendation Results 1

2 MAGIC CLOSET: CLOTHING PAIRING RECOMMENDATION CRITERION Dating Attribute vs. Occasion Model Criterion 1: Wear Properly (occasion-oriented) Attribute vs. Attribute Model T-shirt Occcasion1: Wedding Online shopping websites or the social photos of friends Human Parts Alignment Clothing Attribute Estimation White Sleeveless Strapless Clothing Recommendation Model Clothing Pairing Results (Buy or Borrow) Occcasion2: Funeral RECOMMENDATION CRITERION WHAT TO WEAR (WoW) DATASET Criterion 2: Wear Aesthetically (upper / lower matching) Similar colors: Contrast colors: Flickr, Online shops, and Mechanical Turk Clothing Category Attributes: CLOTHING ATTRIBUTES category Crowdsourcing: Mechanical Turk DETAILED ATTRIBUTES We manually define 6 kinds of clothing attributes based on a comprehensive study of many online shopping websites Upper body Lower body Set Red Orange Yellow Green Blue Purple Blazers Outwear Coats Sweaters GLOBAL UPPER T-shirts Fashion Hoodies Detailed Attribute: Skirts Jeans Shorts Pants Capris Jumpsuits Suits Pants Attribute Name Attibute Values Attribute Name Attibute Values Red Orange Color Pattern Collar Black White Yellow Cotton Chiffon Material Green Brown Gray Silk Woolen Blue Purple Multi-color Denim Leather Vertical Plaid Horizontal Drawing Plain Floral print Strapless V-shape One-shoulder Long Shot Sleeve Jewel Round Shirt collar Sleeveless LOWER Length Long Median Short Dress Suits Skirts Black White Gray Brown Color Vertical Plaid Horizontal Drawing Floral Pattern print Oneshoulder V-shape Multi-color Plain Strapless Jewel Round Shirt Collar collar Chiffon Woolen Cotton Denim Leather Silk Long Short Materials Short Sleeveless Sleeve Median Long Length 2

3 HUMAN POSE ESTIMATION LATENT SVM RECOMMENDATION MODEL POTENTIAL FUNCTION Use the annotated key points in human photos Train one human upper-body detector and one human lower-body detector [1] Potential Function Wear Parameters Properly Attributes Occasion Feature vs. Occasion Potential Feature vs. Attribute Potential Attribute vs. Occasion Potential Attribute vs. Attribute Potential Feature vs. Occasion Potential: Wear Elegantly is the score of the pre-trained multi-class linear SVM. [1] Yi Yang, et.al, Articulated Pose Estimation with Flexible Mixtures of Parts, CVPR 2012 LATENT SVM RECOMMENDATION MODEL POTENTIAL FUNCTION LATENT SVM RECOMMENDATION MODEL FORMULATION Feature vs. Attribute Potential: : confidence score from multiple multi-class SVM classifiers. Each multi-class SVM corresponds to a specific attribute. Formulation Attribute vs. Occasion Potential: : the co-occurrence statistics between the j th attribute and the occasion o. Attribute vs. Attribute Potential: : the co-occurrence statistics of and, the probability of the j th attribute of upper-body clothing taking value while the k th attribute of lower-body clothing taking value. where [1] Pedro Felzenszwalb et.al. A Discriminatively Trained, Multiscale, Deformable Part Model. LATENT SVM RECOMMENDATION MODEL INFERENCE LATENT SVM RECOMMENDATION MODEL INFERENCE Given the parameter model w, we need to solve the following inference problem during recommendation: Clothing Suggestion Criterion: Clothing Pairing Criterion: Given an occasion and a reference coat, we recommend the corresponding trousers/skirts with the largest potential value. Given an occasion, we recommend the clothing pair with the largest potential value. 3

4 (upper) 2013/2/12 RECOMMENDATION MODEL ANALYSIS: Matching Rules Wear Properly The top-3 The top-2 most most representative attributes attributes black wedding plain sleeveless dress silk Funeral long (1) Occasion-Attribute Matching Rules RECOMMENDATION MODEL ANALYSIS: Matching Rules EXPERIMENTAL SETTING (lower) Wear Elegantly Evaluate Criterion: Normalized Discounted Cumulative Gain (NDCG): Long sleeves match long trousers. (2) Attribute-Attribute Matching Rules Fashion hoodies match pants/capris. Baselines: Feature-occasion SVM: predicting occasion from visual features directly without considering attributes. Feature-attribute-occasion SVM: the 1 st layer SVM linearly maps visual features to attribute values; the 2 nd SVM is trained on attribute confidence vectors to predict occasion labels. EXPERIMENTAL SETTING EXP1: CLOTHING SUGGESTION RESULTS Split of the dataset for different experiments. Arrows indicate query vs. repository relationship. The feature-occasion SVM consistently outperforms the feature-attribute-occasion SVM. The proposed latent SVM model outperforms the two baseline models significantly. 4

5 sports Conference Funeral Conference school Dating 2013/2/12 EXP1: CLOTHING SUGGESTION EXEMPLAR RESULTS Higher Score Lower Score Occasion EXP2: CLOTHING PAIRING RESULTS Recommendation Results More Results Personal Album wedding Sports The feature-attribute-occasion SVM significantly performs better than the feature-occasion SVM (=SVM_U + SVM_L). The proposed latent SVM model outperforms the two baseline models significantly. EXP2: CLOTHING PAIRING EXEMPLAR RESULTS Reference Image Recommended Clothing RELATED WORK ACEDEMIC Few researches in the clothing related study: Occasion Score Labeled By Users More Results Street-to-shop clothing retrieval system [1] : Snap a street fashion, and search similar clothing from online shops. Clothing Parsing [2] : Given an image, segment it into several sematic regions, such as trouser, shirts, etc. We are the only work focusing on clothing recommendation task. [1] Si Liu, et.al, Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set, CVPR 2012 [2]Kota Yamaguchi, et.al, parsing clothing in fashion photography, CVPR 2012 RELATED WORK INDUSTRY CONCLUSION: Magic Closet Many fashion sharing websites, where girls learn how to wear aesthetically by looking at others. feedback Social network The first to explore two applications: clothing suggestion and clothing pairing. Construct a large dataset, called What-to-Wear (WoW), containing 24, 417 clothing with complete attribute and occasion annotations. Apply a latent SVM based framework to learn the occasionoriented clothing recommendation model and simultaneously mine the clothing matching rules. tags Great application potential in social media (from social media, and used for social media) 5

6 FUTURE DIRECTIONS Inaccurate body detection and parsing/segmentation Some attribute predictions are not reliable yet very important, such as materials THANK YOU & QA silk cotton cotton cotton Personalized clothing recommendation Complex clothing for Winter Beyond clothing: other wearing and carrying 6