In the domain of audio-visual event perception, which focuses on the temporal localization and classification of events across distinct modalities (audio and visual), existing approaches are constrained by the vocabulary available in their training data. This limitation significantly impedes their capacity to generalize to novel, unseen event categories. Furthermore, the annotation process for this task is labor-intensive, requiring extensive manual labeling across modalities and temporal segments, limiting the scalability of current methods. Current state-of-the-art models ignore the shifts in event distributions over time, reducing their ability to adjust to changing video dynamics. Additionally, previous methods rely on late fusion to combine audio and visual information. While straightforward, this approach results in a significant loss of multimodal interactions. To address these challenges, we propose Audio-Visual Adaptive Video Analysis (AV²A), a model-agnostic approach that requires no further training and integrates an score-level fusion technique to retain richer multimodal interactions. AV²A also includes a within-video label shift algorithm, leveraging input video data and predictions from prior frames to dynamically adjust event distributions for subsequent frames. Moreover, we present the first training-free, open-vocabulary baseline for audio-visual event perception, demonstrating that AV²A achieves substantial improvements over naive training-free baselines. We demonstrate the effectiveness of AV²A on both zero-shot and weakly-supervised state-of-the-art methods, achieving notable improvements in performance metrics over existing approaches.
Overview of the AVVP task. Audio-visual event perception focuses on predicting the temporal boundaries of events within a video that are exclusively visible (shown in blue), exclusively audible (shown in red), or both audible and visible (shown in purple).
Overview of AV2A. The process begins with category selection, where the input video clip passes through a video-level score-level fusion module (blue) to select relevant categories based on a threshold τf. These categories guide segment-level score-level fusion, where a dynamic threshold module (orange) updates thresholds τt via our label-shift technique, using the soft confusion matrix M from prior predictions Y1, . . . , Yt−1 and segment scores P1av, . . . , Pt−1av, cosine similarity between segments and Ptav. Finally, predicted candidates are validated against a confidence threshold τr, retaining only those above it. This figure illustrates the process for audio-visual events; audio and visual events are handled similarly.
Table 1: Comparison of training-free methods on the LLP dataset, reporting AVVP metrics.
Table 2: Comparison with zero-shot methods on AVE datasets, reporting AVEL metrics.
Performance analysis of AV2A, based on the LanguageBind [37] model, showcasing predictions on audio-visual events, specifically those occurring simultaneously in both audio and video. Comparisons highlight (a) improvements and (b) failure-cases relative to state-of-the-art weakly supervised baselines.
@inproceedings{shaar2025adapting,
title={Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds},
author={Shaar, Eitan and Shaulov, Ariel and Chechik, Gal and Wolf, Lior},
booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
pages={3142--3151},
year={2025}
}