Yes, there are several open-source AI models and frameworks being developed that aim to both
describe video content in words and apply sophisticated reasoning to it. These models typically fall under the category of Video-Large Language Models (Vid-LLMs) or multimodal AI.
Key Open-Source Models and Frameworks
video-SALMONN-o1: Described as the "first open-source audio-visual large language model that can perform reasoning to help understand videos better," this project focuses on general video understanding tasks, including the ability to detect synthetic (fake) videos and answer complex questions using step-by-step reasoning. The code and data for the related RivaBench benchmark are available for researchers.
GLM-4.6V: This open-source multimodal model features stronger visual reasoning and long-context understanding, performing global summarization on long videos while retaining the ability to perform fine-grained reasoning on temporal clues.
Univa: An ambitious open-source project aiming to be a "comprehensive video generalist" capable of understanding, editing, and generating complex, long-form video. Early benchmarks suggest its understanding module is highly effective at intricate video tasks.
EgoThinker: This framework endows MLLMs with strong egocentric (first-person view) reasoning capabilities, using spatio-temporal chain-of-thought to understand human intentions and actions in detail. The full code and data are released on GitHub.
SiLVR (Simple Language-based Video Reasoning Framework): This framework is designed for long-video understanding, utilizing a simple, single-pass modular approach to compress video into a language representation and then applying an LLM for reasoning about actions and stories across long horizons.
The Role of Reasoning
The "reasoning" aspect goes beyond simple object detection or action recognition; it involves understanding context, causality, foresight, and implicit intentions, often using "chain-of-thought" processes (breaking down a scenario into steps). Researchers are actively developing benchmarks and training methods to enhance these capabilities in open-source models.
Practical Implementation
To use these models, you would typically need to deploy the machine learning model yourself using open-source libraries like OpenCV or TensorFlow. Many projects release their code and data on platforms like GitHub, allowing developers to access and build upon the research.
Komentarų nėra:
Rašyti komentarą