Beyond Anonymous Captions: Grounding Character Identity in Video Captioning and Question Answering
Beyond Anonymous Captions (BAC) introduces identity-aware video understanding by linking known characters to their locations in the video. This grounding strategy improves captioning and question answering, helping Video-MLLMs reason about both what is happening and who is involved.
Introduction
Video-language models have become remarkably good at describing what happens in a video. Given a movie scene, they can recognize people, clothing, actions, objects, interactions, and even fairly subtle details.
But there is an important difference between:
“A man in a green shirt kisses a woman in a red top while another woman watches from a car.”
and:
“Cam Gigandet, in a green shirt, kisses Minka Kelly, in a red top, while Leighton Meester watches from the car.”
The first caption captures what is happening. The second also captures who is involved.
That distinction is crucial in movies and other narrative videos. Understanding a story is not only about recognizing actions or interactions in isolation; it also requires attaching them to the correct recurring characters across scenes.
This leads to the central question behind Beyond Anonymous Captions (BAC):
If we already know who a character is and where that character appears in the video, how should we communicate this information to a Video-MLLM?
Our key idea is simple: explicitly provide the model with identity-location associations. We first identify known characters and localize them across the sampled frames, then provide this information together with the video. In other words, we tell the model who is where, while leaving the interpretation of the scene to its existing visual reasoning capabilities.
This separation is important. BAC does not attempt to replace the model’s ability to understand actions, appearance, position, or interactions. Instead, it supplies the missing identity link, allowing the model to reason about what each named character is doing, where they are, and who they are interacting with.
In this way, BAC moves video understanding from anonymous people toward identity-aware descriptions and reasoning about recurring characters.
Figure 1. Example of identity-aware video understanding.
Building identity-grounded movie clips
We build our dataset from LSMDC v2, starting from 92 movies and 46,756 video clips.
For each movie, we first retrieve the cast and collect reference images for every actor. These images form a movie-specific reference gallery. We then detect faces in the video, match them to the gallery, and track the identified characters across consecutive frames.
Since the original clips may contain multiple camera cuts, we split them into visually coherent shots. This produces 54,133 shots, with an average duration of about 2.5 seconds.
We then uniformly sample 8 frames per shot and keep shots where at least one identified character appears in the sampled frames. This leaves 31,935 shots from 72 training movies and 6,219 shots from 12 test movies.
At this point, each retained shot contains both the video content and explicit identity-location information: we know who the identified characters are and where they appear across the sampled frames.
Next, we use Gemini 2.5 Flash to generate identity-aware captions for both training and test shots. For the test set, it also generates four person-centric questions per shot and scores the examples for validity, coherence, and identity-grounding difficulty.
We retain challenging, high-quality examples and manually review the captions, questions, and answers. Ambiguous or redundant examples are removed, and overly simple questions are refined.
The final test benchmark contains:
- 750 clips with identity-aware captions
- 3,000 person-centric questions (4 questions per clip)
- 2,074 questions targeting known identities
- 926 UNIDENTIFIED questions
The UNIDENTIFIED cases correspond to questions about visible people whose identity is not provided to the model. They test whether the model can correctly refrain from assigning one of the known character identities.
The questions cover appearance, actions, position, objects, posture, and interactions.
Figure 2 shows the main steps of our pipeline for identity-aware video captioning.
Figure 2: Identity-aware video understanding pipeline
How should we tell the model who is where?
Knowing the identity-location correspondence is only part of the problem. We also need to decide how this information should be presented to the Video-MLLM.
We evaluate five strategies:
Face Textual Coordinates (FTC). The face bounding-box coordinates are provided only in the prompt, while the video frames remain unchanged.
Face Visual Boxes (FVB). A colored bounding box is drawn around each identified face together with its identifier, without providing coordinates in the prompt.
Person Visual Boxes (PVB). The detected face is expanded into an approximate person-level region, which is then visually annotated instead of the face.
Face Visual Boxes + Face Textual Coordinates (FVB+FTC). The model receives both the visual face box and its corresponding coordinates in the prompt.
Person Visual Boxes + Person Textual Coordinates (PVB+PTC). The same combination is applied using the estimated person-level region.
These variants let us ask a simple question: are coordinates alone sufficient, are visual cues more effective, or does combining both provide the strongest grounding?
Experiments
We evaluate our approach on two complementary tasks, identity-aware captioning and person-centric question answering, using the final test benchmark.
To study how the grounding strategy behaves across model scales, we use the Qwen3-VL family at approximately 2B, 4B, and 8B parameters. For each model size, we compare the five grounding representations introduced above under the same evaluation protocol.
This first experiment lets us analyze both which grounding representation works best and how the effect changes as the model becomes larger.
Visual grounding matters
Table 1 summarizes how the five identity-grounding strategies perform across model scales on both identity-aware captioning and person-centric QA.
The first result is clear: textual coordinates alone (FTC) are not enough.
With Qwen3-VL 2B, QA accuracy reaches only 53.87% when identity locations are provided through textual coordinates. Adding visual grounding raises performance to 68.90%. The same pattern holds at larger scales: at 4B, accuracy increases from 62.57% to 79.73%, while at 8B it rises from 59.77% to 88.53%.
A similar behavior is observed for identity-aware captioning, where visual grounding consistently improves both overall caption quality and identity accuracy compared with textual coordinates alone.
Overall, FVB+FTC provides the most consistent performance across both captioning and QA, particularly at the 2B scale. As model size increases, the differences between the visual grounding strategies become smaller, suggesting that larger models can make effective use of a broader range of grounding representations.
Face or full person?
One might expect a full-person box to always be more informative than a face box.
In practice, the answer depends partly on the size of the face.
Although Table 1 shows that FVB+FTC significantly outperforms PVB+PTC at 2B in overall accuracies, a closer look reveals an important exception.
For very small faces, person-level grounding can help: as shown in Table 2, when the target face occupies less than 4,000 pixels², PVB+PTC reaches 91.8% accuracy, compared with 80.0% for FVB+FTC. But when the face is large enough to be visually informative, the more precise face region generally works better.
Large person boxes can overlap with other people and introduce unnecessary visual clutter.
This suggests that the ideal grounding representation may eventually be adaptive: use a larger region when the face is tiny and a precise face region when it is already clearly visible.
Small models also want to name everyone
Another interesting behavior appears when the queried person has no provided identity.
Small models tend to force one of the known identities onto that person.
As shown in Table 1, under the FVB+FTC setting, UNIDENTIFIED accuracy reaches only 16.09% for the 2B model and 49.78% for the 4B model, despite substantially stronger performance on Named questions.
At 8B, this gap nearly disappears (86.93% UNID vs. 88.04% Named), showing that larger models are much better at recognizing when the correct answer is simply UNIDENTIFIED.
Figure 3 shows an example where all three base Qwen3-VL models incorrectly assign the known identity, while the correct answer is UNIDENTIFIED.
Figure 3: UNIDENTIFIED example from Blind Dating (3011_BLIND_DATING_00.50.27.583-00.50.29.145): all three base Qwen3-VL models incorrectly assign the known identity, Chris Pine, to an unidentified person.
Introducing BAC
Grounding already improves off-the-shelf Video-MLLMs considerably.
But we wanted to know whether the model could actually learn this identity-grounding behavior.
We therefore fine-tune Qwen3-VL at three scales using LoRA:
- BAC-2B
- BAC-4B
- BAC-8B
The models are trained on approximately 32K identity-aware captioned movie shots.
Importantly, they receive captioning supervision only. They are never trained on QA task.
Yet the learned identity-grounding behavior transfers strongly to question answering.
BAC improves both captioning and QA
BAC improves both captioning and QA at every scale, as shown in Table 3, with particularly strong QA gains for smaller models. BAC-2B improves QA accuracy from 68.90% to 80.33%, BAC-4B from 79.73% to 90.13%, and BAC-8B reaches 93.20%.
Importantly, BAC is trained only on identity-aware captions, yet these gains transfer strongly to QA. This effect is especially pronounced on UNIDENTIFIED questions, where the base models struggle: BAC improves UNIDENTIFIED accuracy from 16.09% to 45.46% at 2B, 49.78% to 76.13% at 4B, and 86.93% to 91.68% at 8B.
Smaller BAC models can beat much larger VLMs
We also compare BAC with other Video-MLLM families at similar sizes.
As shown in Table 4, BAC is the strongest model in every parameter group.
More surprisingly, the gains from task-specific adaptation are large enough to compensate for substantial differences in model size.
BAC-2B reaches 80.33% QA accuracy, slightly above the strongest evaluated 4B base model.
BAC-4B reaches 90.13%, outperforming every evaluated 8–9B base model.
And BAC-8B reaches 93.20%.
This led us to ask a more difficult question:
How do the identity-specialized BAC models compare with frontier VLMs?
BAC models vs. frontier models
Table 5 compares BAC with much larger open-weight and proprietary models using the same identity-grounded inputs.
BAC-8B ranks behind only GPT-5.6 Sol in our evaluation, while outperforming Qwen3-VL-235B-A22B, Gemini 2.5 Flash, and Claude Sonnet 4.6. BAC-4B also surpasses the Qwen3-VL-235B-A22B model, while BAC-2B outperforms Claude Sonnet 4.6.
These results show that specialized identity-aware adaptation can outperform much larger general-purpose models on identity-grounded video reasoning.
Qualitative examples
1) THE_ROOMMATE_00.23.30.000-00.23.35.938
2) THE_ROOMMATE_00.22.07.000-00.22.13.798
3) Juno_01.24.32.310-01.24.37.291
Figure 4. Qualitative identity-aware QA examples. Identity-mapping colors match the bounding-box colors in each frame. (1) BAC correctly predicts UNIDENTIFIED; (2) BAC correctly identifies the queried named person; (3) BAC fails to identify the queried named person, while only GPT-5.6 Sol succeeds. * GPT-5.6 Sol with thinking.
Discussion and limitations
Our experiments suggest a simple conclusion:
Video-MLLMs reason much better about character identity when we explicitly communicate who is where.
Textual coordinates alone provide a relatively weak spatial signal. Visual identity grounding is much easier for the models to exploit, while combining visual boxes and textual coordinates provides particularly consistent performance.
The optimal grounding region is not always the same either. Person-level grounding can be useful for tiny faces, while face-level grounding becomes more reliable when the face is sufficiently visible.
Task-specific adaptation substantially improves performance on both captioning and QA, even though BAC is trained only for caption generation.
There are still several limitations.
Our experiments focus on movie footage, where identities come from a predefined cast. Extending the same setting to unconstrained real-world video would introduce additional identity, privacy, and ethical challenges.
Character identification is also performed upstream. Errors in face recognition or tracking can therefore propagate to the Video-MLLM.
Finally, some spatial and relational questions remain challenging even for BAC-8B. Position and interaction reasoning are notably harder than appearance-based questions.
Conclusion
Standard video captioning tells us what is happening.
Identity-aware video understanding additionally requires knowing who is doing it.
Our results show that these two capabilities can be connected surprisingly effectively by providing Video-MLLMs with explicit identity-linked spatial grounding.
And with lightweight LoRA adaptation, BAC turns this grounding signal into a more general identity-aware video representation: BAC-8B reaches 93.20% QA accuracy, while BAC-4B surpasses substantially larger general-purpose models.
Rather than redesigning the Video-MLLM architecture, BAC shows that a relatively simple combination of character identification, spatial grounding, and task-specific adaptation can go a long way toward answering the more interesting question:
Who is doing what, where, and with whom?
A similar approach to BAC is already integrated into MXT-3 and available to Moments Lab customers, bringing identity-aware video understanding into real-world editorial workflows.
Paper: https://arxiv.org/abs/2610.10163
Code, benchmark, training data, and checkpoints: https://github.com/momentslab/beyond-anonymous-captions
