DeCoRe

Scientific description

Deep convolutional neural networks ( CNNs) and recurrent neural networks ( RNNs) have recently led to major advances in various fields, including object recognition, machine translation, and speech recognition. One of the key distinguishing features of these approaches across different application domains is their ability to learn end-to-end. 

Conventional methods generally involve a signal preprocessing step during which features are extracted, such as MFCC (Mel Frequency Cepstral Coefficients) for speech or SIFT (scale-invariant feature transform) for images. In deep, end-to-end trainable systems, each processing layer (from the raw input signal upward) involves trainable parameters that allow the system to learn the most appropriate features. DeCoRe brings together researchers from LJK, GIPSA-LAB, and LIG in computer vision, machine learning, speech, natural language processing, and information retrieval to foster collaborative research in this rapidly evolving field, with the aim of supporting future advances in these research areas over the next decade.

Two doctoral students are fully funded by DeCoRe. One is working on the automatic generation of natural-language image descriptions. The other is working on techniques for object recognition and localization in contexts involving numerous object classes (>1,000) and continuous data streams for training.

Results

Significant progress has recently been made in the automatic generation of natural language image captions using encoder-decoder models. These models encode the image content into a vector representation using a convolutional neural network. A recurrent neural network decoder uses this vector representation to sequentially generate a natural language sentence based on the image’s content. We have developed a model based on this encoder-decoder paradigm, which sequentially processes different regions of the image as the sentence is generated (see figure below). 

This allows us to generate better image captions, as local visual features can be linked to the words in the caption, rather than having to convey all visual information through a single overall image representation. This work has been submitted for publication [3].

In another study, we proposed a neural network for multilingual annotation projection in natural language processing [5] (see figure below).

Coordinators

Laurent Besacier (LIG)

Denis Pellerin (GIPSA-lab)

Georges Quénot (LIG)

Jakob Verbeek (Inria/LJK)

Valuation

The research conducted as part of DeCoRe has already resulted in two submissions to the 2017 CHIST-ERA call for proposals on lifelong learning systems:

• LLAMA: Lifelong learning approaches for multimedia analysis.
• ADEL2: Autonomous driving made easier through lifelong learning.

Notable publications

[1] A. Berard, O. Pietquin, C. Servan, and L. Besacier. “Listen and translate: A proof of concept for end-to-end speech-to-text translation using a
.” In NIPS Workshop on End-to-End Learning for Speech and Audio Processing, 2016.

[2] P. Luc, C. Couprie, S. Chintala, and J. Verbeek. “Semantic segmentation using adversarial networks.”
In NIPS Workshop on Adversarial Training, 2016.

[3] M. Pedersoli, T. Lucas, C. Schmid, and J. Verbeek. Areas of attention for image captioning. arXiv:1612.01033.

[4] S. Saxena and J. Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems (NIPS) 29, 2016.

[5] O. Zennaki, N. Semmar, and L. Besacier. “Inducing multilingual text analysis tools using bidirectional recurrent neural networks.” In COLING, 2016.