EUSIPCO 2025Audio-Video Vector Alignment

Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio–Video Foundation Model

Ali Vosoughi*University of Rochester

* Work completed during an internship at Microsoft Research, Redmond, WA, USA.

Read the paper Explore AVE-2 Cite this work

Abstract

Integrating audio and visual data for training multimodal foundational models remains a challenge. The Audio-Video Vector Alignment (AVVA) framework addresses this by considering AV scene alignment beyond mere temporal synchronization, and leveraging Large Language Models (LLMs) for data curation. AVVA implements a scoring mechanism for selecting aligned training data segments. It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure with contrastive learning on AV pairs. Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture and data curation approach. AVVA achieves a significant improvement in top-k accuracies for video-to-audio retrieval on all datasets compared to DenseAV, while using only 192 hrs of curated training data. Furthermore, an ablation study indicates that the data curation process effectively trades data quality for data quantity, yielding increases in top-k retrieval accuracies on AudioCaps, VALOR, and VGGSound, compared to training on the full spectrum of uncurated data.

From raw pairs to training data

How the curation works

Describe the audio and video separately, then reason about how they fit together. Language guides data selection; AVVA learns directly from the selected audio–video pairs.

The AVVA framework: LLaVA-Next-V describes video and LTU-AS describes audio. Mistral scores alignment and selects training pairs for Whisper and DINOv2 encoders with an InfoNCE objective. — Overview of the proposed audiovisual alignment approach. (a) Our method's data curation stage uses multimodal reasoning to retain only highly aligned data. It uses LLaVA-Next-Video for video reasoning, LTU-AS for audio processing, and Mistral for alignment scoring. (b) AVVA employs Whisper (audio), DINOv2 (video backbone), without the need for textual mediation during training.
View the full-size framework ↗

Segment the video
Split input videos into 3-second clips, keeping the audio paired with its video.
Describe each modality
LLaVA-Next-Video captions the video. LTU-AS describes the audio, including speech and environmental sounds.
Score the alignment
Mistral compares the descriptions using five criteria: Temporal Alignment, Spatial Coherence, Contextual Relevance, Physical Causality, and Sound Source Visibility. Each score is on a scale of 0–10.
Select, then train
Average the scores with equal weights and apply a selection threshold. Train AVVA on the retained pairs using Whisper and DINOv2 with contrastive learning.

See what the pipeline scores

Examples

These four clips are a small sample drawn from one shard of the AVE-2 train split, shown to illustrate the released fields. They are part of the data, not the whole release: AVE-2 is metadata-first, so the captions, scores and rationales are shown here and the underlying YouTube media is not redistributed.

Real scored segments from AVE-2, the dataset that followed this work. Captions, audio descriptions, and scores below are machine-generated annotations, not human-verified labels.

High alignment

An acoustic guitar, heard and seen

-fUNb4eiu1w ↗
Segment 02 · 3–6 s in AVE-2
Metadata source

Video caption · opening paragraph

Read the full video caption

The video features a person playing an acoustic guitar in what appears to be a casual indoor setting. The individual is wearing a dark t-shirt with a star emblem on the chest and is seated, holding the guitar with both hands. The background includes a chair and some items that suggest a personal space, such as clothing hanging on a rack or shelves. The scene transitions smoothly from one frame to another without significant changes in the visual elements. There are no visible texts or subtitles within the frames provided. The lighting is consistent throughout the video, indicating it was taken during the day under natural light conditions.

Audio description

Labels with acoustic features: Rich and full -> Musical instrument; Full of various frequencies -> Music

Pipeline scores · 0–10
Temporal Alignment	10
Spatial Coherence	10
Contextual Relevance	10
Physical Causality	9
Sound Source Visibility	10

The video caption describes a person playing an acoustic guitar; the audio description identifies music and a musical instrument, and the visible-source field identifies the acoustic guitar.

Read the pipeline’s reasoning

Temporal Alignment. The audio description of the music perfectly aligns with the visual of the person playing an acoustic guitar.
Spatial Coherence. The audio description of the rich and full musical instrument matches the visual of the acoustic guitar being played.
Contextual Relevance. The audio description of the music and the visual of the person playing an acoustic guitar both suggest a casual indoor setting.
Physical Causality. The audio description of the music can logically be attributed to the action of the person playing the acoustic guitar.
Sound Source Visibility. The acoustic guitar is the only visible active sound source in the scene.

High alignment

A telephone and its ringing bell

-07O11QVLxo ↗
Segment 02 · 3–6 s in AVE-2
Metadata source

Video caption · opening paragraph

Read the full video caption

The video features a vintage rotary dial telephone placed on a desk. The background is dimly lit, with various papers and books scattered around the area. The phone has a blue base with a silver dial and red buttons. There are no visible texts or subtitles in the video. The scene transitions smoothly from one frame to another without any significant changes in the objects or their positions. The lighting remains consistent throughout the video, casting soft shadows on the desk surface. No other actions or events occur within this single frame of the video.

Audio description

Labels with acoustic features: High-pitched and piercing -> Alarm; High pitched and short -> Telephone bell ringing; Rich in frequency modulation -> Speech; High-pitched and beeping -> Alarm clock; Reverberant and echoey -> Inside, large room or hall; Clear and distinct -> Speech

Pipeline scores · 0–10
Temporal Alignment	10
Spatial Coherence	10
Contextual Relevance	8
Physical Causality	10
Sound Source Visibility	10

The caption describes a rotary telephone, the audio description includes telephone bell ringing, and the visible-source field identifies the telephone.

Read the pipeline’s reasoning

Temporal Alignment. The audio description of the telephone bell ringing aligns perfectly with the ringing sound heard in the video.
Spatial Coherence. The high-pitched and short sounds described as a telephone bell ringing are coming from the visible telephone in the video, which is the only potential sound source in the scene.
Contextual Relevance. The telephone ringing in the audio provides a relevant context for the vintage rotary dial telephone depicted in the video.
Physical Causality. The telephone ringing in the audio can be logically attributed to the visible telephone in the video, as this is the only object capable of producing such a sound in the scene.
Sound Source Visibility. All active sound sources are visible in the video, including the telephone. No objects in the video are silent that could potentially produce sound.

Low alignment

Film credits, a barking dog

2UQi-0zeQsI ↗
Segment 01 · 0–3 s in AVE-2
Metadata source

Video caption · opening paragraph

Read the full video caption

The video displays a list of names and titles, likely from a film or television show credits. The background is a maroon color with text in yellow and white fonts. The top section lists the director's name, followed by the animation team members, including Marvin Woodward, George Kreisl, Gerry Hathcock, Sandy Strother, Macdonald MacDonald, Jack Huber, Oliver Wallace, Karl Karpe, and Art Landy. The scene transitions between different scenes, each with its own title and characters listed below it. For example: - "Direction" - Marvin Woodward - "Animation" - George Kreisl - "Story" - Sandy Strother - "Music Layout" - Oliver Wallace - "Background" - Art Landy The overall setting appears to be a movie or TV show set in an office environment, as suggested by the presence of multiple screen shots and the formal presentation style of the credits.

Audio description

Labels with acoustic features: Rich in frequency modulation -> Speech; High-pitched and barking -> Dog

Pipeline scores · 0–10
Temporal Alignment	0
Spatial Coherence	0
Contextual Relevance	0
Physical Causality	0
Sound Source Visibility	0

The video caption describes film or television credits, while the audio description identifies speech and a barking dog. All five released scores are zero; the source list identifies the dog as invisible and the credits as silent.

Read the pipeline’s reasoning

Temporal Alignment. There is no direct correlation between the audio (a dog barking) and the video (credits of a film or television show).
Spatial Coherence. The video is a static list of credits, while the audio features a dog barking, which does not have a specific spatial location within the video.
Contextual Relevance. The video is related to film or television credits, while the audio is a dog barking, which has no apparent connection to the context of the video.
Physical Causality. The video does not depict any objects or actions that could logically produce the sound of a dog barking.
Sound Source Visibility. The video does not depict any objects or characters that could be the source of the dog barking sound.

Off-screen sound source

Bird calls outside the frame

-7QTaeEKaig ↗
Segment 01 · 0–3 s in AVE-2
Metadata source

Video caption · opening paragraph

Read the full video caption

The video depicts a cartoon scene featuring a character lying in bed with a telephone on the side. The background is plain and light-colored, which contrasts with the dark blue of the bed and the white of the pillow. There are no other objects or characters visible in the frame. Objects: - A bed - A telephone - A pillow Actions/Events: - The character appears to be sleeping or resting. - The telephone is connected to the bed via a cord. - The character's position suggests they might be waking up or preparing for sleep. Temporal Information: - The video starts with the character in bed. - It transitions to the character sitting up in bed. - The scene then shifts to the character looking at something off-screen while still in bed. Visual Characteristics: - The color scheme is simple, with the bed being the most prominent object due to its size and placement. - The lighting is even, without any strong shadows or highlights that would suggest a specific time of day or mood. - The overall atmosphere is calm and serene, typical of a peaceful bedroom setting. Spatial Layout: - The bed occupies the central part of the image. - The telephone is positioned to the left of the bed, close to the viewer. - The character is seated upright, facing away from the camera. Text and Graphics: - No text or subtitles are present in the visual content. Emotional Tone: - The mood of the video seems tranquil and relaxed, consistent with a quiet, restful environment. Contextual Inference: - Based on the visual cues provided by the scene, it can be inferred that this is a moment captured during a routine activity such as getting ready for bed or taking a nap.

Audio description

Labels with acoustic features: High-pitched and melodious -> Bird; High pitched and melodic -> Bird vocalization, bird call, bird song

Pipeline scores · 0–10
Temporal Alignment	3
Spatial Coherence	0
Contextual Relevance	4
Physical Causality	5
Sound Source Visibility	0

The video caption describes a cartoon character resting in bed with a telephone and pillow. The audio description identifies bird calls, and invisible_active_sources explicitly identifies those calls. The released visibility score is zero, with varied scores on the other four criteria.

Read the pipeline’s reasoning

Temporal Alignment. The bird calls do not seem to correspond with any specific action or event in the video, as the character is shown to be resting or sleeping.
Spatial Coherence. There is no spatial coherence between the audio and video as the bird calls are not coming from any visible or logical source within the video.
Contextual Relevance. While the bird calls may symbolize nature or the outdoors, there is no indication in the video to suggest that the scene is set in such an environment.
Physical Causality. The bird calls are audible, which implies that there is a source for the sounds, but it is not present or visible within the video.
Sound Source Visibility. There are no visible active sound sources in the video that could be producing the bird calls.

Selected examples from the training split. Segment times are local AVE-2 excerpt windows, not timestamps in the original YouTube video. Expand each video caption to read it in full. Captions and scores are reproduced without edits; example titles and selection notes are editorial. Annotations reproduced with attribution to Ali Vosoughi and co-authors under CC BY 4.0. See the access and citation terms.

Evidence from the paper

Results

The complete main retrieval table, including every comparison method and both retrieval directions.

AVVA results use an MRE threshold of 7.6 out of 10. A→V denotes audio-to-video retrieval; V→A denotes video-to-audio retrieval.

Performance on Audio (A) - Video (V) Retrieval (Top-k={1,3,10}) (%). Standard deviations shown as superscripts depict performance variation over K=100 retrieval repetitions of random test subsets of size N=100.
Method	Retrieval Type	AudioCaps			VALOR			VGG-Sound
Method	Retrieval Type	Top1	Top3	Top10	Top1	Top3	Top10	Top1	Top3	Top10
Wav2CLIP	A→V	1.20^{± 0.98}	6.40^{± 3.01}	18.60^{± 3.83}	3.60^{± 0.80}	8.60^{± 0.49}	18.20^{± 3.54}	3.40^{± 1.36}	8.20^{± 1.17}	19.80^{± 3.06}
Wav2CLIP	V→A	3.80^{± 2.14}	10.00^{± 3.22}	20.00^{± 3.63}	4.20^{± 2.64}	8.00^{± 4.24}	19.00^{± 3.52}	3.80^{± 1.94}	9.20^{± 2.32}	19.60^{± 1.62}
Random	A→V	1.40^{± 0.49}	3.80^{± 0.75}	11.80^{± 1.17}	1.20^{± 0.75}	3.20^{± 0.40}	11.60^{± 1.62}	1.20^{± 0.40}	3.40^{± 0.49}	11.60^{± 2.15}
Random	V→A	1.00^{± 0.00}	3.60^{± 0.80}	10.80^{± 1.17}	1.20^{± 0.40}	3.20^{± 0.75}	11.00^{± 0.63}	1.00^{± 0.00}	3.00^{± 0.00}	10.60^{± 0.80}
DenseAV	A→V	10.20^{± 2.04}	22.60^{± 4.54}	49.40^{± 4.54}	7.80^{± 5.19}	19.00^{± 5.90}	41.80^{± 4.79}	6.80^{± 2.64}	16.00^{± 2.90}	43.20^{± 3.43}
DenseAV	V→A	1.40^{± 0.80}	5.60^{± 1.85}	26.40^{± 2.73}	2.20^{± 1.17}	5.80^{± 2.79}	24.60^{± 7.68}	1.60^{± 1.02}	5.00^{± 0.63}	22.60^{± 2.58}
ImageBind	A→V	62.00^{± 2.28}	83.40^{± 3.01}	92.60^{± 1.85}	55.80^{± 4.66}	71.60^{± 3.61}	85.00^{± 3.74}	50.60^{± 3.14}	74.00^{± 5.93}	88.20^{± 2.99}
ImageBind	V→A	64.00^{± 5.37}	85.40^{± 4.27}	95.40^{± 0.80}	58.80^{± 4.71}	73.60^{± 4.36}	86.60^{± 3.20}	53.20^{± 3.31}	73.40^{± 6.02}	85.60^{± 3.20}
AVVA (Ours)	A→V	6.57^{± 2.30}	13.84^{± 2.80}	31.68^{± 3.57}	6.69^{± 2.13}	15.63^{± 3.52}	33.67^{± 4.40}	6.71^{± 1.91}	15.02^{± 2.73}	33.86^{± 4.23}
AVVA (Ours)	V→A	6.23^{± 2.09}	14.70^{± 3.17}	31.06^{± 3.52}	7.75^{± 2.61}	16.64^{± 3.65}	34.27^{± 4.71}	6.86^{± 2.34}	14.47^{± 3.15}	32.84^{± 3.89}

Scroll across the table to see all datasets. Values and wording are from the paper.

Cross-modal retrieval

AVVA achieves audio-to-video accuracy comparable to DenseAV, with significantly improved video-to-audio accuracy, despite using only 192 hrs of carefully curated data compared to DenseAV's 5,800 hrs, demonstrating a 30x improvement in data efficiency.

Data curation

A key observation from our experiments is that increasing the amount of training data does not always lead to better performance.

Temporal alignment

The analysis reveals a clear peak in similarity at 0 sec shift between audio and video, providing evidence of meaningful audio-video learning.

The dataset that followed

Dataset

AVE-2 ↗

AVE-2 provides audiovisual captions, alignment scores, score rationales, and sound-source annotations. Access is gated: requests are reviewed individually and require agreement to the dataset’s citation terms.

The annotation layer is attributed to Ali Vosoughi and co-authors under CC BY 4.0. Research using AVE-2 should cite both the AVVA paper and Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs (preprint), as requested on the dataset card.

Access and citation notice

AVE-2 is released for research use. By requesting access you agree to the following: (1) you will cite the AVE-2 papers listed on this card in any paper, model, dataset, demo or report that uses AVE-2 or anything derived from it; (2) the annotation layer is used under CC BY 4.0 with attribution to Ali Vosoughi and co-authors; (3) the underlying audiovisual media is obtained under its own terms and is not redistributed with this release; (4) the metadata is not redistributed without this notice and the citation requirement; (5) access requests are reviewed individually and may be declined.

Continuing the research

Follow-on work

Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs ↗

SoundCLIP is the follow-on preprint. Its project page presents the token-substitution study and links to the current paper and the AVE-2 dataset.

SoundCLIP project Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs (preprint)

Reference this work

Citation

@inproceedings{vosoughi2025quality,
  title = {Quality Over Quantity? {LLM}-Based Curation for a
           Data-Efficient Audio--Video Foundation Model},
  author = {Vosoughi, Ali and Emmanouilidou, Dimitra and Gamper, Hannes},
  booktitle = {EUSIPCO 2025},
  year = {2025},
  url = {https://eurasip.org/Proceedings/Eusipco/Eusipco2025/pdfs/0000286.pdf}
}

Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio–Video Foundation Model

Abstract

How the curation works

Segment the video

Describe each modality

Score the alignment

Select, then train

Examples

An acoustic guitar, heard and seen

A telephone and its ringing bell

Film credits, a barking dog

Bird calls outside the frame

Results

Cross-modal retrieval

Data curation

Temporal alignment

Dataset

AVE-2 ↗

Follow-on work

Projected Audio Tokens Gain Retrieval and Lose Grounded Generation in Multimodal LLMs ↗

Citation