Transfer Learning and Visualization of Neural Networks for Artistic Images

Transfer learning from large-scale natural image datasets, particularly ImageNet, fine-tuning standard deep convolutional neural network models and using the corresponding pre-trained network have become the de facto method for art analysis applications. Nevertheless, there are large differences in dataset sizes, image style and task specifications between natural image classification and the target artistic images, and there is little understanding of the effects of transfer learning. In this work, we explore some properties of transfer learning for artistic images. We compared different ways to obtain an image classifier: fine-tuning, or not, of pre-trained models and training models from scratch. We also use feature visualization techniques in order to understand more precisely what the network learned on those specific artistic datasets. Those visualization of deep neural networks internal representations can help to highlight how neural networks build up their "understanding" of images. We observed that the network could specify some pre-trained filters in order to adapt them to the new modality of images. On the other hand, the network can also learn new, highly structured filters specific to artistic images when the lower-level layers of the initial model are "frozen". In particular, it is possible to obtain classifiers with equivalent classification performances but with different hidden representations, that can be specific to artistic images or not.

Nicolas Gonthier, Yann Gousseau & Saïd Ladjal (Télécom Paris)

Projet MemoRekall