Scientific
Publications
SimulU: Training-free Policy for Long-form Simultaneous Speech-to-Speech Translation
Amirbek Djanibekov, Luisa Bentivogli, Matteo Negri, Sara Papi
Simultaneous speech-to-speech translation (SimulS2S) is essential for real-time multilingual communication, yet it remains underexplored: existing methods rely on heavy training and operate only on short, pre-segmented utterances that do not generalize well to continuous speech. To close this gap, we introduce SimulU, the first training-free policy for long-form SimulS2S. SimulU uses history-management and speech-output-selection strategies that leverage the cross-attention of pre-trained end-to-end models to control both input history and output generation. On MuST-C across 8 languages, SimulU delivers a better or comparable quality-latency trade-off against strong cascaded models. By removing the requirement for specialized training, SimulU offers a viable route toward end-to-end SimulS2S in realistic, long-form settings.
Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Bogdan Kostić, Conor Fallon, Julian Risch, Alexander Löser
The rapid advancement of Large Language Models (LLMs) has established standardized evaluation benchmarks as the primary instrument for model comparison. Yet, their reliability is increasingly questioned due to sensitivity to shallow variations in input prompts. We investigate how truth-conditionally equivalent lexical and syntactic perturbations affect benchmark scores, evaluating 23 modern LLMs across three benchmarks (MMLU, SQuAD, and AMEGA). Using two linguistic pipelines, one for synonym substitution and another applying dependency parsing for syntactic transformations, we generate meaning-preserving variations of the inputs. Lexical perturbations consistently induce substantial, statistically significant performance degradation, whereas syntactic changes produce more heterogeneous effects that occasionally improve results; both can destabilize model leaderboards on complex tasks. Robustness does not consistently scale with model size and is strongly task-dependent, suggesting models rely more on surface-level lexical cues than on genuine linguistic understanding. We argue that robustness testing should become a standard component of LLM evaluation.
SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs
Niccolo Avogaro, Nayanika Debnath, Li Mi, Thomas Frick, Junling Wang, Zexue He, Hang Hua, Konrad Schindler, Mattia Rigotti
Test-time scaling, dynamically expanding the token budget during inference, remains fragile for vision-language models because unstructured reasoning chains entangle perception and reasoning, letting small perceptual mistakes cascade into completely wrong answers; reasoning also demands costly reinforcement learning with hand-crafted rewards. We introduce SPARC, a modular framework that explicitly decouples visual perception from reasoning. Inspired by sequential sensory-to-cognitive brain processing, SPARC uses a two-stage pipeline: the model first localizes question-relevant regions through explicit visual search, then reasons over those regions to reach a final answer. This supports independent test-time scaling with asymmetric compute allocation (for example, prioritizing perception under distribution shift), enables selective optimization of just the perceptual stage when it limits performance, and allows compressed contexts by running global search at lower resolutions and reserving high-resolution processing for selected regions, cutting visual token count and compute. SPARC outperforms monolithic baselines and strong visual-grounding approaches, improving Qwen3VL 4B on the V* VQA benchmark by 6.7 points and surpassing thinking with images by 4.6 points in an out-of-distribution setting with a 200x lower token budget.
GeoFAR: Geography-Informed Frequency-Aware Super-Resolution for Climate Data
Chang Xu, Gencer Sumbul, Li Mi, Robin Zbinden, Devis Tuia
Super-resolving climate data is crucial for fine-grained decision-making in various domains, ranging from agriculture to environmental conservation. However, existing super-resolution approaches struggle to generate the high-frequency spatial information present in climate data, especially over regions showing complex terrain variability. A key obstacle lies in a frequency bias existing in both deep neural networks (DNNs) and climate data: DNNs exhibit such bias by overfitting to low-frequency information, which is further exacerbated by the prevalence of low-frequency components in climate data (e.g., plains, oceans). As a consequence, geography-dependent high-frequency details are hard to reconstruct from coarse climate inputs with DNNs. To improve the fidelity of climate super-resolution (SR), we introduce GeoFAR: by explicitly encoding climatic patterns at different frequencies, while learning implicit geographical neural representations (i.e., related to location and elevation), our approach provides frequency-aware and geography-informed representations for climate SR, thereby reconstructing fine-grained climate information at high resolution. Experiments show that GeoFAR is a model-agnostic approach that can mitigate high-frequency prediction errors in both deterministic and generative SR models, demonstrating state-of-the-art performance across various spatial resolutions, atmospheric variables, and downscaling ratios.
MIAM: Modality Imbalance-Aware Masking for Multimodal Ecological Applications
Robin Zbinden, Wesley Monteith-Finas, Gencer Sumbul, Nina van Tiel, Chiara Vanalli, Devis Tuia
Multimodal learning is crucial for ecological applications, which rely on heterogeneous data sources (e.g., satellite imagery, environmental time series, tabular predictors, bioacoustics) but often suffer from incomplete data across and within modalities (e.g., unavailable satellite image due to cloud cover, missing records in a time series). While data masking strategies have been used to improve robustness to missing data by exposing models to varying input subsets during training, existing approaches typically rely on static masking and inadequately explore the space of input combinations. As a result, they fail to address modality imbalance, a critical challenge in multimodal learning where dominant modalities hinder the optimization of others. To fill this gap, we introduce Modality Imbalance-Aware Masking (MIAM), a dynamic masking strategy that: (i) explores the full space of input combinations; (ii) prioritizes informative or challenging subsets; and (iii) adaptively increases the masking probability of dominant modalities based on their relative performance and learning dynamics. We evaluate MIAM on two key ecological datasets, GeoPlant and TaxaBench, with diverse modality configurations, and show that MIAM significantly improves robustness and predictive performance over previous masking strategies. In addition, MIAM supports fine-grained contribution analysis across and within modalities, revealing which variables, time segments, or image regions most strongly drive performance.
An Efficient and Effective Encoder Model for Vision and Language Tasks in the Remote Sensing Domain
João Daniel Silva, João Magalhães, Devis Tuia, Bruno Martins
The remote sensing field has recently adopted approaches based on Large Vision and Language Models (LVLMs) capable of handling several tasks that combine computer vision and natural language processing, with significant effort going toward gathering large training datasets for remote-sensing-specific tasks such as image captioning and visual question answering. However, the cost of using and training LVLMs is high due to their large number of parameters, and even with parameter-efficient adaptation methods the computational demands may remain prohibitive for most institutions. To address this, we investigate encoder-only architectures and propose a compact model designed for multi-task learning, targeting task combinations rarely handled by a single unified model: generating text from remote sensing images and cross-modal retrieval. Our model, GeoMELT (named from Multi-task Efficient Learning Transformer), achieves benchmark results that demonstrate both its effectiveness and its efficiency.
BOOM: Beyond Only One Modality — KIT's Multimodal Multilingual Lecture Companion
Sai Koneru, Fabian Retkowski, Christian Huber, Lukas Hilgert, Seymanur Akti, Enes Yavuz Ugan, Alexander Waibel, Jan Niehues
The spread of global education and online learning has made content localization an important challenge. Because lecture materials mix audio and visual slides, systems need to handle several input types, and effective translations should preserve all modalities so learners get text for reading, slides for visual understanding, and speech for auditory learning. We present BOOM, a multimodal multilingual lecture companion that jointly translates lecture audio and slides. It produces synchronized outputs in three forms: translated text, localized slides that preserve visual elements, and synthesized speech, letting students access lectures in their own language while retaining the complete original content. We further show that slide-aware transcripts yield cascading benefits for downstream tasks such as summarization and question answering. A demo video and code are released under the MIT License.
How to Evaluate Speech Translation with Source-Aware Neural MT Metrics
Mauro Cettolo, Marco Gaido, Matteo Negri, Sara Papi, Luisa Bentivogli
Automatic evaluation of speech-to-text translation (ST) has traditionally relied on reference-based machine translation (MT) metrics that ignore the source speech. Source-aware neural MT metrics, which condition on the source, promise more accurate assessment but are hard to apply to ST because the source is audio rather than text. We study how to use source-aware MT metrics for ST evaluation, comparing source text obtained from automatic speech recognition (ASR) against back-translations of the reference, together with a two-step cross-lingual re-segmentation procedure that aligns hypotheses and references. Across 79 ST systems, ASR-based source is preferable when the word error rate is below 20%, while back-translations always represent a computationally cheaper but still effective alternative. These robustness findings are confirmed on a low-resource language pair (Bemba-English) and validated against human quality judgments. The cross-lingual re-segmentation algorithm enables robust use of source-aware MT metrics in ST evaluation, paving the way toward more accurate and principled evaluation methodologies for speech translation.
A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results
Thai-Binh Nguyen, Katerina Zmolikova, Pingchuan Ma, Ngoc Quan Pham, Christian Fuegen, Alexander Waibel
We introduce Multi-Modal Context-Aware Recognition (MCoRec), a new task in the ninth CHiME Challenge that tackles the cocktail-party problem of overlapping conversations in a single room by drawing on audio, visual, and contextual information. The dataset features natural, unscripted group conversations with speech overlap reaching up to 100% and fragmented turns, and systems must determine who speaks when, what, and with whom by transcribing each speaker and grouping them into their respective conversations. Audio-only baselines exceed a 100% word error rate, while adding visual cues delivers roughly a 50% improvement, demonstrating the value of multi-modality. We also cover the task's motivation, the data collection methodology, and the baseline systems built for MCoRec.
From Slides to Chatbots: Enhancing Large Language Models with University Course Materials
Tu Anh Dinh, Philipp Nicolas Schumacher, Jan Niehues
Large Language Models have progressed quickly and can help students learn, though earlier research found they still struggle to give accurate answers in university-level computer science courses. We study whether adding university course materials improves LLM performance. A central difficulty is that course materials such as lecture slides and transcripts differ from standard text: slides include visuals such as images and formulas, while transcripts consist of spoken, loosely structured language. We compare two approaches for injecting course-specific knowledge, Retrieval-Augmented Generation (RAG) and Continual Pre-Training (CPT), and for slides we additionally test a multi-modal RAG method that shows retrieved material to the generator as images. Because course materials are relatively small in size, RAG is more effective and efficient than CPT, and including slides as images in the multi-modal setup meaningfully boosts results over text-only retrieval. These findings offer practical guidance for building AI assistants that support learning and teaching, and we hope to inspire similar work in other educational contexts.
Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks
Supriti Sinhamahapatra, Jan Niehues
Leading ASR systems mainly use acoustic data and overlook other multi-modal context, even though visual information helps with disambiguation and adaptation. Most prior work targets speaker images for noise handling; we additionally integrate presentation slides for scientific talks. Our approach has three parts: we build a benchmark for multi-modal presentations that includes analysis of transcribing specialized terminology; we investigate ways to enhance speech models with multi-modal signals, addressing dataset scarcity through data augmentation; and we train a model on the augmented data. This results in a relative reduction in word error rate of approximately 34% across all words and 35% for domain-specific terms compared to the baseline model.
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Fevziye Irem Eyiokur, Dogucan Yaman, Hazım Kemal Ekenel, Alexander Waibel
Embodied Reference Understanding (ERU) requires locating a target object using both language and pointing cues. Earlier methods struggle in ambiguous situations with multiple candidate objects. We propose a new ERU framework that combines LLM-based data augmentation, a depth-map modality, and a depth-aware decision module. This design enables robust integration of linguistic and embodied cues, improving disambiguation in complex or cluttered environments. Experiments on two datasets show that our method significantly outperforms existing baselines, achieving more accurate and reliable referent detection.