梦幻故事:通过LLM指导的多主题一致传播的开放域故事可视化
IEEE transactions on pattern analysis and machine intelligence
|August 19, 2025
概括
DreamStory使用大型语言模型 (LLM) 和新型扩散模型生成一致的多主题故事可视化. 这个框架确保了主题的外观和属性在生成的场景中的一致性.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 当前的故事可视化方法在生成的图像序列中与主题一致性作斗争.
- 扩散模型显示出有希望的结果,但对于多主体叙述缺乏连贯性.
- 自动,开放域的故事可视化仍然是一个重大挑战.
研究的目的:
- 介绍DreamStory,这是一个自动框架,用于生成连贯的,多主题的故事可视化.
- 为了应对从文本叙事中获得的图像序列中保持主体一致性的挑战.
- 开发一种新的多主题一致传播模型,以增强视觉讲故事的功能.
主要方法:
- 利用大型语言模型 (LLM) 作为故事导演来生成描述提示和注释主题.
- 使用一种新型的多主体一致扩散模型 (MSD) 结合了掩盖的相互自我注意 (MMSA) 和掩盖的相互交叉注意 (MMCA).
- 采用多模式 (主体肖像和文本描述) 引导MSD进行一致的多主体生成.
主要成果:
- 梦幻故事成功地产生了视觉上引人注目的和连贯的故事可视化,并具有一致的多主体表示.
- 与MMSA和MMCA一起的MSD有效地确保了对象的详细外观和语义一致性.
- 建立了一个新的基准,DS-500,以评估故事可视化框架的性能,主题识别和生成一致性.
结论:
- DreamStory提供了一种有效的解决方案,用于自动,开放域的故事可视化,并改进了多主题一致性.
- 拟议的MSD架构显著提高了从文本中视觉讲故事的质量和连贯性.
- 该DS-500基准方便未来的研究和评估在故事可视化领域.
相关概念视频
Diffusion
198.9K
Diffusion is the passive movement of substances down their concentration gradients—requiring no expenditure of cellular energy. Substances, such as molecules or ions, diffuse from an area of high concentration to an area of low concentration in the cytosol or across membranes. Eventually, the concentration will even out, with the substance moving randomly but causing no net change in concentration. Such a state is called dynamic equilibrium, which is essential for maintaining overall...
198.9K
Assessment of Diffusion and Perfusion
1.1K
Understanding and evaluating diffusion and perfusion is critical in assessing a patient's respiratory and circulatory health. These processes play key roles in maintaining the body's internal environment, ensuring that tissues receive adequate oxygen while waste products are efficiently removed.
The Role of Diffusion in Respiration
Diffusion is the process by which molecules move from an area of higher concentration to an area of lower concentration. In the respiratory system, this...
The Role of Diffusion in Respiration
Diffusion is the process by which molecules move from an area of higher concentration to an area of lower concentration. In the respiratory system, this...
1.1K
Facilitated Diffusion
609
The plasma membrane, a critical structure in cellular biology, houses an array of transporters, or carrier proteins, interspersed within its lipid bilayer. These proteins play a crucial role in solute transport through facilitated diffusion, a form of passive diffusion that uses transporters to move the molecules across the membrane.
In this process, substrates such as organic compounds and ions interact with a transporter on one side, triggering conformational changes in proteins that enable...
In this process, substrates such as organic compounds and ions interact with a transporter on one side, triggering conformational changes in proteins that enable...
609


