PMRAG · Dynamic Planning and Multimodal Faithful Generation in MRAG Systems for Complex Information Access

This project develops dynamic planning methods and faithful multimodal generation for Retrieval-Augmented Generation systems (MRAG). It addresses the limitation of the current MRAGs that primarily retrieve text and often produce ungrounded or hallucinated multimodal outputs when asked to combine images, tables, code, and video. I will design a hybrid multimodal retriever that indexes diverse content types, a learned planning module that decides which modalities and retrieval depth to use for each query, and constrained generation techniques that enforce grounding to retrieved evidence. The work combines scalable representation learning, cross-modal alignment, and reinforcement learning for planning under budget and fidelity constraints. Benchmarks will be created by extending existing datasets with multimodal references and factuality labels; evaluation will measure retrieval precision, generation faithfulness, and utility on downstream tasks (QA, table reasoning, code synthesis, and multimodal summarisation). The project includes research secondments with an industrial partner to transfer methods to real-world search and digital-assistant applications. Training will focus on multimodal ML, reproducible research, and ethics. Hosted by the University of Amsterdam under the supervision of Dr Mohammad Aliannejadi, this fellowship will advance methods for reliable multimodal information access and prepare me for a research career at the intersection of IR and generative AI.