Multimodal representation learning
I work on models that map images and text into a shared embedding space, so that one can be used to search for the other. At Amazon Visual Search (A9) I led training of two generations of these models.
CLIP model for visual search
I led training of the team’s internal CLIP model: contrastive image-text representation learning at CLIP scale, used for multimodal retrieval.
- Role: led model training
- Team: Amazon Visual Search (A9)
- Status: shipped worldwide, in multiple Amazon production features
General-purpose VLM embedding model
I led training of a general-purpose embedding model built on a vision-language model.
- Role: led model training
- Team: Amazon Visual Search (A9)
- Status: rolling out to production
Articles
Bringing multimodality to Amazon visual search system
Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2024
Pure image-to-image matching in visual search suffers from false positives caused by matching local visual patterns. This paper adds image-text alignment losses, drawing on vision-language pretraining, as constraints on the image-to-image matching loss in deep metric learning, and applies the approach to the Amazon visual search system.
Leveraging Large Language Models for Multimodal Search
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024
Multimodal search lets people combine an image, for fine-grained product detail, with text, for modifications. Natural-language queries are often ambiguous, implicit or partly irrelevant, so this paper introduces a multimodal search system that uses large language models for query parsing and rewriting.