Content Based Video Narration of Gameplay with Vision Language Models

My arXiv paper on generating spoken esports-style gameplay commentary with a vision-language model and text-to-speech — no game-specific code required.

My paper “Content Based Video Narration of Gameplay with Vision Language Models” is now on arXiv: arXiv:2608.14016 (submitted 14 August 2026).

What the paper covers

The paper presents a system that generates spoken, esports-style commentary for gameplay videos using a vision-language model and text-to-speech, with no game-specific code.

• Temporal mosaic packing — nine uniformly sampled frames are arranged into a single 3×3 image, enabling motion reasoning with much smaller image payloads.
• Context-conditioned prompting — recent narrations are replayed back to the model so it stops repeating itself.
• Duration-conditioned generation with elastic audio alignment — commentary is generated to fit a target duration, then time-aligned to the segment.
• Cloud and local inference — supports both cloud TTS and a local 4-billion parameter text-to-speech model running on Apple devices.

The paper also includes qualitative analysis and is honest about the limitations: hallucinated game states, and prosody artifacts introduced by time-scaling the audio.

Related project writeup: Content-Based Video Narration Using Deep Learning.