IMPLEMENTATION OF ARTIFICIAL NEURAL NETWORKS IN MULTIMODAL IMAGE RECOGNITION AND TEXT.

Deep networks are already applied successfully to unsupervised feature learning for single modalities (e.g., text, images or audio). At First, rapid progress carried out in object detection has identi?ed models that ef?ciently identify and label multiple regions of an image. Secondly, recent advances in image captioning have expanded the complexity of the label space from a permanent set of categories to sequence of words able to express signi?cantly richer concepts. Here, we propose a unique application of deep networks to learn features over multiple modalities (image to text) i.e. image captioning. In this model of image captioning, Convolutional Neural Networks is applied to multiple regions of images followed by bidirectional Recurrent Neural Networks applied to sentences. Thus, the two sub-networks, CNN and RNN interact with each other in a multimodal layer to form the entire m-RNN model.This model is capable of learning long-term interactions. This arises from using a repeated visual memory that learns to reconstruct the visual features as new words are read or generated.