My mission is to conduct research to improve the robustness and reliability of AI systems. My contributions towards this goal are in the domains of machine learning, computer vision, and natural language processing.
I work with Yezhou Yang and Chitta Baral at ASU, and closely collaborate with Rushil Anirudh at Lawrence Livermore National Laboratories. I received my MS in ECE from Carnegie Mellon University, where I worked with Aswin Sankaranarayanan.
My domain expertise lies in "semantic vision", i.e. computer vision tasks that seek to assign "meaning" to what we see -- this includes "designative" tasks such as image classification; "communicative" tasks involving both vision and language such as visual question answering, visual reasoning, and image captioning.
The main focus of my Ph.D. is on robust visual understanding, to address problems such as domain shift/out-of-distribution generalization, linguistic robustness (logical, semantic), visual robustness (corruptions, geometric transformations, attribute-level shift).
A Research Statement and a poster summarizing some of my work, is now available. (presented at CVPR DC 2022)
Collaboration/Mentorship Opportunities: If you're a PhD student interested in collaborating with me on robust machine learning in Vision/NLP/V+L (domain generalization, adversarial attack/defense etc.), or other related topics, please send me an email (if you're at ASU, we can discuss it over coffee). I'm always happy to dish out advice and share my experiences w.r.t. admissions to Ph.D. programs in CS/EE/CE (help me help you by sending me a list of specific questions over email).
23 Jun 2022 Presented my work at CVPR Doctoral Consortium at CVPR 2022
20 Jun 2022 Organized the 1st Workshop on Open-Domain Retrieval Under Multi-Modal Settings (O-DRUM) at CVPR 2022
25 Apr 2022 Recognized as Highlighted Reviewer for ICLR 2022 (top ~8%)
ALT discovers diverse and adversarial transformations using an image-to-image neural network with learnable weights. ALT improves the state-of-the-art single domain generalization performance on three benchmarks and is significantly better than pixel-wise adversarial training and standard data augmentation techniques.
SDRO: a distributed robust optimization method that operates with linguistic transformations of sentence inputs, SISP: a suit of semantics-inverting (SI) and semantics-preserving (SP) linguistic transformations, and an ensembling technique for vision-and-language inference.
In this work, we conduct a comprehensive study of common data modification strategies and evaluate not only their in-domain and OOD performance, but also their adversarial robustness (AR). This work serves as an empirical study towards understanding the relationship between generalizing to unseen domains and defending against adversarial perturbations.
We seek to improve information retrieval (IR) using neural retrievers (NR) in the biomedical domain, using a three-pronged approach. (1) a template-based question generation method, (2) two novel pre-training tasks that are closely aligned to the downstream task of information retrieval, (3) the ``Poly-DPR'' model which encodes each context into multiple context vectors.
VQA models trained with two additional objectives: object centroid estimation and relative position estimation, lead to improved performance on spatial reasoning questions (in GQA) in fully supervised and few shot settings as well as improved O.O.D. generalization.
We show that models can be trained without any human-annotated Q-A pairs, but only with images and associated text captions. Our experiments suggest gains on benchmark with shifted priors (VQA-CP) over baselines which use full supervision from human-authored QA data.
Scene completion from sparse and incomplete label maps. `Halluci-Net' is a 2-stage method that captures the object co-occurrence relationships, to produce dense label maps from incomplete labelmaps and object boundaries, for image synthesis.
Unsupervised Reading Comprehension method that operates directly on a single test passage. Synthetic QA pairs are generated from the passage, and models are trained on these. When a new human-authored test question appears, models infer answers better than previous unsupervised methods.
An adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to attributes-space. Studies robustness to semantic shifts that are beyond L-p norm perturbations, on 3 types of naturally occurring perturbations --- object-related shifts, geometric transformations, and common image corruptions.
MUTANT is a training paradigm that exposes VQA models to perceptually similar, yet semantically distinct mutations of the input image or question. We use a pairwise consistency loss between answers to original and mutant inputs as a regularization, along with an answer embedding NCE loss. MUTANT establishes a new SOTA (+10%) on the VQA-CP challenge (for generalization under Changing Priors)
Actions in videos are inherently linked to latent social and commonsense aspects. We present the first work on generating commonsense captions directly from videos, to describe latent intentions, attributes, and effects of humans in videos. Additionally we explore the use of open-ended video-based commonsense question answering (V2C-QA) as a way to enrich our captions.
VQA models struggle at negation, antonyms, conjunction, disjunction! We show a capability of answering logically composed questions with our novel modules and datasets, while retaining performance on VQA data.
Given two images (source, target) with different object configurations, what is the sequence of steps to re-arrange source to match target? For this reasoning task, our modular approach that contains a visual encoder and an event-sequencer/planner, and exhibits inductive generalization.