Sitemap
A list of all the posts and pages found on the site. For you robots out there, there is an XML version available for digesting as well.
Pages
publications
Indoor scene layout estimation from a single image
Published in 24th International Conference on Pattern Recognition (ICPR), 2018
A deep learning approach to estimating the layout of an indoor scene from a single image in real time, combining a fully convolutional network, a layout-degeneration augmentation method and a new training pipeline.
AugGAN: Cross Domain Adaptation with GAN-based Data Augmentation
Published in Proceedings of the European Conference on Computer Vision (ECCV), 2018
A structure-aware image-to-image translation network, with encoders, generators, discriminators and parsing nets for both domains in one framework. It preserves image objects and keeps translations consistent, so it can generate large-scale training data for a new domain.
GAN-Based Day-to-Night Image Style Transfer for Nighttime Vehicle Detection
Published in IEEE Transactions on Intelligent Transportation Systems, 2020
GAN-based data augmentation for training CNN detectors across a large domain shift: AugGAN transforms on-road driving images from day to night while preserving the objects in them, to improve nighttime vehicle detection.
Leveraging Large Language Models for Multimodal Search
Published in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
Multimodal search lets people combine an image, for fine-grained product detail, with text, for modifications. Natural-language queries are often ambiguous, implicit or partly irrelevant, so this paper introduces a multimodal search system that uses large language models for query parsing and rewriting.
Bringing multimodality to Amazon visual search system
Published in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2024
Pure image-to-image matching in visual search suffers from false positives caused by matching local visual patterns. This paper adds image-text alignment losses, drawing on vision-language pretraining, as constraints on the image-to-image matching loss in deep metric learning, and applies the approach to the Amazon visual search system.
research
Multimodal representation learning
Embedding models that put images and text in one space, from a CLIP model shipped worldwide in Amazon production features to a general-purpose embedding model built on a vision-language model.
Training at scale
The infrastructure that makes large multimodal models trainable: an internal distributed training platform on Ray that delivered two generations of VLM-based models.
Agentic AI
Agents that do real work and whose work can be checked: fine-tuning for agentic orchestration of visual search, and verification-first agents for IC physical design, the idea I’m now pursuing as a founder.
Domain adaptation and scene understanding
Earlier computer vision work: GAN-based image-to-image translation that generates training data for a new domain, such as day to night for vehicle detection, and real-time indoor layout estimation from a single image.