Domain adaptation and scene understanding
Before working on multimodal models, I worked on computer vision for domains where labelled training data is scarce.
GAN-based data augmentation across domains
Detectors trained on one domain, such as daytime driving scenes, often fail on another, such as night. AugGAN is a structure-aware image-to-image translation network that transforms images to the target domain while preserving the objects in them, so the translated images can be used as training data. We applied it to day-to-night style transfer for nighttime vehicle detection.
Indoor scene layout estimation
A real-time approach to estimating the layout of an indoor scene from a single image, built on a fully convolutional network with a layout-degeneration augmentation method.
Articles
GAN-Based Day-to-Night Image Style Transfer for Nighttime Vehicle Detection
IEEE Transactions on Intelligent Transportation Systems, 2020
GAN-based data augmentation for training CNN detectors across a large domain shift: AugGAN transforms on-road driving images from day to night while preserving the objects in them, to improve nighttime vehicle detection.
AugGAN: Cross Domain Adaptation with GAN-based Data Augmentation
Proceedings of the European Conference on Computer Vision (ECCV), 2018
A structure-aware image-to-image translation network, with encoders, generators, discriminators and parsing nets for both domains in one framework. It preserves image objects and keeps translations consistent, so it can generate large-scale training data for a new domain.
Indoor scene layout estimation from a single image
24th International Conference on Pattern Recognition (ICPR), 2018
A deep learning approach to estimating the layout of an indoor scene from a single image in real time, combining a fully convolutional network, a layout-degeneration augmentation method and a new training pipeline.