Illustration2Vec: A Semantic Vec

Introduction

In this paper, we propose (1) a vector representation for illustrations, which takes the semantic difference into consideration, and (2) a novel image exploration tool for finding reference illustrations.

Related Work

相似度学习
卷积神经网络

Our approach

Dataset
We first build a database consisting of approximately 1.3 million illustrations and the associated tags

Layer configuration
输入图像预测所属的tag
这样同一tag的图像就会有类似的隐层激活状态
我们的网络见下图右侧

Binary hashing and nearest neighbor search
We first insert an additional sigmoid layer with 4,096 units after the last convolutional layer, and retrain the entire CNN initialized with the pre-trained network by using the training dataset. Then, we binarize the activation of the sigmoid layer with a threshold of 0.5 and regard it as a 4,096-bit hash string.

Evaluation and Application

Tag prediction

Nearest neighbor search

t-SNE