联合空间时空模型 (JMSC):用于视听学习的联合时空模型与语义补充
Xinfu Xu1, Fan Yang1, Zhibin Yu1,2
1Faculty of Information Science and Engineering, Ocean University of China, Qingdao 266100, China.
Sensors (Basel, Switzerland)
|February 27, 2026
概括
这项研究引入了一个新的视听学习框架 (JMSC),通过整合声音和视觉来增强场景理解. JMSC 改进了语义理解和动态视频分析,有效地实现了最先进的结果.
科学领域:
- 计算机视觉 计算机视觉
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 视听学习整合了听觉和视觉线索,以获得全面的场景理解.
- 以前的方法通常涉及计算上昂贵的完整模型微调.
- 参数高效的调方法越来越多地用于适应大型模型的视听任务.
研究的目的:
- 解决在音频和视觉模式之间利用互补语义方面的挑战.
- 提高对动态视频背景的理解,包括不断变化的空间和时间属性.
- 为高级视听学习提出一个新的框架,即带有语义完成的联合时空建模 (JMSC).
主要方法:
- JMSC采用跨模态潜伏重建来深入整合来自不同模式的语义信息.
- 它鼓励从掩盖的对应物中重建一个模式的摘要,超越浅层的相关性.
- 该框架在音频指导下共同建模空间属性和时间变化,以实现统一表示.
主要成果:
- 在多个下游视听任务中,JMSC实现了最先进的性能.
- 该方法证明了改进的语义理解和精确的定位/跟踪在动态场景.
- 与现有方法相比,JMSC保持了较高的计算效率.
结论:
- JMSC为整合视听信息提供了有效的解决方案,克服了先前方法的局限性.
- 该框架增强了对复杂动态视频内容的理解.
- JMSC为视听学习提供了一个计算高效和高性能的方法.
相关概念视频
Associative Learning
1.6K
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
1.6K
Chunking and Rehearsal in Sensory Memory
640
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
640
Observational Learning
1.1K
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
1.1K
Elaborative Rehearsals
423
Elaborative rehearsal is a crucial cognitive strategy that strengthens information encoding in long-term memory by making meaningful connections between new data and pre-existing knowledge. This approach contrasts with maintenance rehearsal, which involves simple repetition without delving into the significance of the information. While maintenance rehearsal might temporarily keep information active in short-term memory, it is less effective for long-term retention.
The effectiveness of...
The effectiveness of...
423
Cognitive Learning
1.5K
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
1.5K
Purposive Learning
545
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
545

