Originally proposed in natural language processing, transformers are attracting growing interest in computer vision, providing state-of-the-art results for tasks such as image classification or object detection. In this talk, I present the underlying motivation and the basic architecture of Vision Transformers (ViT). I detail their difference with classical convolution -based architectures for classification, and more general framework for different tasks in computer vision. I also present ViT pre-training with large multimodal language and vision datasets, for downstream tasks with few or zero-shot supervision.

Dense prediction tasks, such as semantic segmentation, are nowadays tackled with data-hungry deep learning architectures. However, oftentimes only limited data is available. In this talk, we argue the need for versatility of deep neural architectures from various perspectives.
First, we discuss the federated learning (FL) paradigm to train deep architectures in a distributed setting with data available only at remote clients. We address non-i.i.d. distribution of samples among clients via 1) a naïve FL optimizer that is fair from the users' perspective, and 2) a prototype-guided FL optimizer that is evaluated also on FL segmentation benchmarks.
Second, we briefly overview model adaptation to unseen visual domains with no ground truth annotations available and we discuss a recent synthetic dataset (SELMA) to aid the segmentation task on such domains.
Finally, we empower deep models to recognize novel semantic concepts without forgetting previously learned ones. We investigate continual semantic segmentation via knowledge distillation, latent space regularization, and replay samples retrieved via weakly-supervised GANs or web-crawled images.

Augmented reality applications have rapidly spread across online retail platforms and social media, allowing consumers to virtually try-on a large variety of products, such as makeup, hair dying, or shoes. However, parametrizing a renderer to synthesize realistic images of a given product remains a challenging task that requires expert knowledge. While recent work has introduced neural rendering methods for virtual try-on from example images, current approaches are based on large generative models that cannot be used in real-time on mobile devices. This calls for a hybrid method that combines the advantages of computer graphics and neural rendering approaches. In this paper, we propose a novel framework based on deep learning to build a real-time inverse graphics encoder that learns to map a single example image into the parameter space of a given augmented reality rendering engine. Our method leverages self-supervised learning and does not require labeled training data, which makes it extendable to many virtual try-on applications. Furthermore, most augmented reality renderers are not differentiable in practice due to algorithmic choices or implementation constraints to reach real-time on portable devices. To relax the need for a graphics-based differentiable renderer in inverse graphics problems, we introduce a trainable imitator module. Our imitator is a generative network that learns to accurately reproduce the behavior of a given non-differentiable renderer. We propose a novel rendering sensitivity loss to train the imitator, which ensures that the network learns an accurate and continuous representation for each rendering parameter. Automatically learning a differentiable renderer, as proposed here, could be beneficial for various inverse graphics tasks. Our framework enables novel applications where consumers can virtually try-on a novel unknown product from an inspirational reference image on social media. It can also be used by computer graphics artists to automatically create realistic rendering from a reference product image.

In this talk after a few words about Naver and Naver Labs, I will shortly present a few recently published works from our lab related to Self-supervision such as MoCHi Mixing of Contrastive Hard Negatives and ICMLM (Image-conditioned Masked Language Modelling) and to continual Domain adaptation such as CDAML (Continual DA with Meta-Learning) and OASiS (Online Adaptation for Semantic Image Segmentation).

Due to the rise of deep learning and the accessibility of big human-annotated datasets, tremendous progress has been made in the fundamental computer vision task of semantic segmentation. However, because each pixel of the image needs to have a label, annotations are quite expensive. As a result, the annotation cost hinders the applications of semantic segmentation in the real world. In the presentation, we outline ways that significantly lower the cost by utilizing less expensive and more readily available annotations.
We first look into the use of partial annotations, where labels are only given for specific areas of the image. We begin with an incremental learning application, where the objective is to expand a model to learn new classes without forgetting and without being given annotations for existing classes. We present a straightforward adjustment of the cross-entropy loss to deal with this situation. The proposed losses are then extended to the point-and-scribble supervised segmentation, where only a small portion of the image's pixels are annotated.
Finally, we show the scenario of not having any pixel-level information. The goal is to learn a segmentation model using cheap and widely available image-level labels, that only indicate the presence of an object in the image without providing any localization cue. We review the current state-of-the-art and illustrate the current solutions based on Class-Activation Maps. Then, extending these techniques, we introduce a framework that learns to segment new classes over time from image-level labels.

I'll talk about future prediction in video sequences. We propose to address the inherent uncertainty in future predictions with stochastic models. While most of the previous methods predict the future in the pixel space, we propose to predict the future also in the motion space to separately model appearance and motion history. We then extend our solution to real-world driving scenarios where the background moves according to the ego-motion of the vehicle. We predict the changes in the static part by modeling the structure and ego-motion. Conditioned on the static prediction, we predict the remaining changes in the dynamic part which correspond to independently moving objects. Finally, we propose to combine information from multiple cameras into a Bird’s Eye View (BEV) representation and predict the future in that compact representation. We efficiently learn the temporal dynamics in the BEV representation with a state space model. Our models outperform the previous methods on standard future frame prediction datasets MNIST, KTH, and BAIR but especially in real-world driving datasets KITTI, Cityscapes, and NuScenes.