Identify clothes you are wearing from images of people who are nice fashion with deep learning and make it searchable

Fashion coordination application "iQON(Icon) "to develop and operateVASILYWe are developing a unique system that can detect and search items worn in snapshots. If this is implemented in a shopping site, for example, you may be able to imitate the entire coordination and purchase only items you like, just by uploading your favorite snapshot.

Detection and retrieval of fashion items by deep learning - VASILY DEVELOPERS BLOG
http://tech.vasily.jp/entry/detection_and_retrieval

When the system loads snapshots, it detects fashion items from photos and searches for similar items.


Roughly speaking, it seems to be divided into the following three tasks, and VASILY uses two kinds of networks to achieve this.

· Detect the area corresponding to the fashion item from the photo
· Forecast categories of detected fashion items
Search items that are similar to the detected fashion items from the DB

◆ Fashion item detection and category prediction
For fashion item detection and category prediction, "Single Shot MultiBox Detector(SSD) "is used as a method.

The network structure of the SSD is as follows. It seems to be advantageous to use the output (feature map) of each convolution layer piled up in layers to predict the area and predict the category.


Code of SSD is published on GitHub and development staff of VASILY recommends "Use this code as it is when you want to try SSD." Fine tuning experiment requires operation not written in README Although it may become, since the author responds with a common pitfall in the issue, if you are in trouble, please try looking for an issue. "

GitHub - weiliu89 / caffe at ssd


◆ Search for fashion items
Then, "similarity learning" or "feature extraction" is used for fashion item search.

When searching for an item, use the image of the item (trimmed snapshot) and extract various features such as item color and shape from this image. Then, searching for similar items by similarity calculation is a mechanism to search for items similar to the items detected. However, the key to searching for items is the snapshot (trimmed image), so various lights and backgrounds are included. In VASILY, we take an approach to acquire a function that returns a large value if it is the same item regardless of this noise by learning. It seems that triplet loss which is learning similarity of deep learning system was adopted for this learning.

◆ How can you actually search for similar items?
Even if difficult words are lined up, there are many people who are stumpy champagne. You can see what kind of items are detected from the snapshot using the actually developed system and what kind of items are similar as the similar items by looking at the image below.

Items detected from snapshots are three tops, bags, pants. Overall, the shape of items are listed from similar to completely different, but the color is pretty much the impression that items close to the original item are displayed as similar items.


As for the result of the second sheet, it is about the level that I think that "the same item as the original snapshot is displayed as a search result?"


In addition, the developer of VASILY says, "We have implemented several experiments and it is now implemented, but I feel there is still room for further advancement.With aiming for a more user-friendly system, I would like to work on expanding the search target. "