Video LLMs answer questions about a video by encoding patches from its frames into visual tokens, but a video produces so many patches that serving these models at high throughput is expensive. Prior pruners reduce the number of patches by removing redundancy without considering the question, or by selecting frames and patches separately. In video question answering, background regions and events unrelated to the question are unnecessary, so their patches can be pruned. A patch is needed only when it shows a relevant region during a relevant event, so its spatial and temporal relevance should be decided together. We present Puget, an efficient patch pruner that considers both the spatial and the temporal relevance of each patch to the question. Spatially, Puget discards patches that are visually irrelevant to the question. Temporally, it discards patches from events unrelated to the question, e.g., the end of the video when the question asks about its beginning. Instead of relying only on heuristics or proxy objectives, Puget learns its pruning preference from the video LLM it prunes for. We train Puget in three steps, with spatial pretraining on image segmentation datasets turned into questions by prompt templates, temporal pretraining on video QA datasets with temporal labels distilled from a large reasoning model, and preference fine-tuning on spatial and temporal labels verified by the target video LLM. On 4 video QA benchmarks with Qwen3.5-9B, Puget matches the accuracy of prior pruners with up to 7.3× lower post-upload time to first token, and it improves accuracy by up to 9.7 points over Vista-LLM at matched time to first token.
@misc{kittivorawong2026puget,
title = {Puget: A Spatiotemporal-Relevance Patch
Pruner for High-Throughput Video LLMs},
author = {Chanwut Kittivorawong and Litao Hu and
Zhaoning Yu and Ruixue Lian and
Xuewen Zhang and Xiaohui Chen and
Hao Wang and Zihao Wang and
Aashu Singh and Fujun Liu and
Alvin Cheung},
year = {2026},
}