A Spatiotemporal-Relevance Patch Pruner
for High-Throughput Video LLMs

Chanwut (Mick) Kittivorawong1, Litao Hu2, Zhaoning Yu2, Ruixue Lian2, Xuewen Zhang2, Xiaohui Chen2, Hao Wang2, Zihao Wang2, Aashu Singh2, Fujun Liu2, Alvin Cheung1
1University of California, Berkeley    |    2Meta

Video LLMs answer questions about a video by encoding patches from its frames into visual tokens, but a video produces so many patches that serving these models at high throughput is expensive. Prior pruners reduce the number of patches by removing redundancy without considering the question, or by selecting frames and patches separately. In video question answering, background regions and events unrelated to the question are unnecessary, so their patches can be pruned. A patch is needed only when it shows a relevant region during a relevant event, so its spatial and temporal relevance should be decided together. We present Puget, an efficient patch pruner that considers both the spatial and the temporal relevance of each patch to the question. Spatially, Puget discards patches that are visually irrelevant to the question. Temporally, it discards patches from events unrelated to the question, e.g., the end of the video when the question asks about its beginning. Instead of relying only on heuristics or proxy objectives, Puget learns its pruning preference from the video LLM it prunes for. We train Puget in three steps, with spatial pretraining on image segmentation datasets turned into questions by prompt templates, temporal pretraining on video QA datasets with temporal labels distilled from a large reasoning model, and preference fine-tuning on spatial and temporal labels verified by the target video LLM. On 4 video QA benchmarks with Qwen3.5-9B, Puget matches the accuracy of prior pruners with up to 7.3× lower post-upload time to first token, and it improves accuracy by up to 9.7 points over Vista-LLM at matched time to first token.

@misc{kittivorawong2026puget,
      title         = {Puget: A Spatiotemporal-Relevance Patch
                              Pruner for High-Throughput Video LLMs}, 
      author        = {Chanwut Kittivorawong and Litao Hu     and
                       Zhaoning Yu           and Ruixue Lian  and
                       Xuewen Zhang          and Xiaohui Chen and
                       Hao Wang              and Zihao Wang   and
                       Aashu Singh           and Fujun Liu    and
                       Alvin Cheung},
      year          = {2026},
}