Real-world video is continuous, long, sparse in useful information, and often shaped by long-range causal relationships. The basic processing paradigm of current VLMs, however, is still better suited to static images or short video clips. The central tension is that a model must preserve a small amount of critical information across videos that may last tens of minutes, hours, or even arrive as continuous streams, and later retrieve, combine, and reason over that information correctly. At the same time, it is impossible to process every frame losslessly and with equal weight.

The most direct approach is to extend the context length. The key question is whether more frames can simply be fed into the model. Common strategies include transferring long-context capabilities from LLMs to vision, or performing end-to-end modeling through long-sequence parallelism and positional-encoding extension. Representative examples include LongVA, LongVILA, Gemini 1.5, and Qwen2.5-VL. This family of methods is straightforward and intuitive, but its weakness is that context utilization remains low, and positional-encoding extension can introduce clear positional bias.

The second common approach is compression and sparsification. Since long-context utilization is still inefficient, compression and sparsity aim to make better use of a limited context budget. The core problem is how to preserve key spatiotemporal information under an extremely small token budget. Typical methods include keyframe or key-clip selection, hierarchical compression, KV sparsification, and query-aware token retention. Examples include LVChat, SlowFast-LLaVA, HICom, Video-XL, VideoChat-Flash, Video-XL-Pro, and Video-XL-2. These methods are relatively easy to implement and suitable for engineering deployment, but they can lose details during compression, and the compression loss is usually irreversible. For query-aware methods, another issue is strong dependence on the given question. Therefore, reversible compression and evidence reconstruction remain major bottlenecks for this direction.

Beyond changing the model itself or compressing tokens, some recent work has started to explore external extension mechanisms for increasingly long videos. One important direction is to build external memory for streaming video. The system processes the video stream online, writes past information into a memory bank, and then retrieves or updates that memory according to the question. This addresses the problem of preserving historical event states across minutes or hours. Representative methods include MovieChat, MA-LMM, VideoStreaming, VideoLLM-online, VideoChat-Online, Hour-LLaVA, and WorldMM. These systems are closer to realistic streaming scenarios and naturally support long-form input, but they also make write and retrieval policies difficult to design. When external memory is compressed, memory summaries may also become distorted. The main challenges for this line of work are how to jointly store visual evidence and textual summaries, and how to coordinate memory at multiple scales.

In addition to long-sequence memory, retrieval and reasoning are also central problems. Some methods use RAG and agents to solve the problem of finding the right evidence in a long video before reasoning over it. They convert long videos into documents, graphs, subtitles, OCR results, or object indexes, then retrieve evidence before answering, or let an agent call tools step by step. Representative methods include DrVideo, Video-RAG, VideoAgent, GroundVQA, and ViG-RAG. This direction has low training cost, stronger interpretability, and makes it easier to add tools, but it also suffers from brittle pipelines and weak retrieval robustness. A difficult open question is whether the retrieval target should be a frame, clip, scene, object, or event graph.

To further improve the accuracy and reliability of video understanding systems, temporal sensitivity and evidence grounding have become important research directions. A model should not only answer a question, but also point to when, where, or which segment provides the evidence. Common ideas include timestamp modeling, localization heads, grounding constraints, and structured evidence outputs, producing methods such as TimeChat and GroundVQA. These methods are closer to real verifiable systems, but annotation is expensive, evaluation is complex, and joint training for open-ended answers and precise grounding is still immature.

Overall, mainstream work has moved from clip-level video LLMs toward system-level long-video pipelines. Sampling, compression, memory, retrieval, training data, and evaluation protocols have become more important than simply optimizing the visual encoder or projector. System parallelism and streaming encoding are also becoming core contributions. In addition, memory is moving from implicit token caches toward explicit, structured, and multimodal designs. Early methods such as MovieChat, MA-LMM, and PMB mainly relied on a single memory bank. Hour-LLaVA then began to adaptively integrate question-related and spatiotemporal information from cached full-video context. More recent systems such as WorldMM further separate episodic, semantic, and visual memory, and use an adaptive agent to perform iterative retrieval across multiple scales. This means that the structure and control strategy of memory itself has become one of the most important problems in long-video understanding.