InstructSee: ダイナミッククエリ生成による指示認識およびフィードバック駆動マルチモダル検索

Guihe Gu^1,2,3, Yuan Xue^1,2,3, Zhengqian Wu^1,2,3

¹National Engineering Research Center for Multimedia Software (NERCMS), Wuhan 430072, China.

Sensors (Basel, Switzerland)

|August 28, 2025

まとめ

この研究は,クロスモダル検索のための命令認識フレームワークを導入し,ビジュアル言語の調整のための複雑な命令からユーザの意図をよりよく理解するために,大規模な言語モデル (LLM) を強化します.

科学分野:

人工知能
コンピュータ・ビジョン
自然言語処理

背景:

クロスモダルの検索は,特に画像とテキストを並べて,異なるデータタイプを橋渡しすることを目的としています.
現在の方法は,複雑なまたは進化するユーザー指示を正確に検索する上で課題に直面しています.
マルチモダルのクエリにおける暗黙のユーザの意図を理解することは,依然として重要な研究ギャップです.

研究の目的:

新しいクロスモダルの表現学習の枠組みを開発する.
自然言語の命令からユーザの意図を把握するシステムの能力を高める.
マルチモダルの検索システムの適応性と精度を向上させる.

主な方法:

命令認識のダイナミッククエリ生成メカニズムを提案した.
大型言語モデル (LLM) の統合された意味論的推論能力
指令とユーザーフィードバックに基づいて動的に構築され,繰り返し精製されたクエリ表現.

主要な成果:

このフレームワークは,標準のベンチマークの検索精度と適応性を大幅に改善しました.
既存の固定クエリベースラインを上回った.
クロスモダルの調整と一般化能力の向上を証明した.

結論:

提案されたフレームワークは,暗黙のリクエストの意図を効果的に推測し,適応します.
LLMで拡張されたダイナミッククエリ生成は,複雑な命令ベースのクロスモダルのリトリーチャーに不可欠です.
この方法は,より直感的で正確なマルチモダルの情報にアクセスするための有望な方向性を提供します.

キーワード:

クロスモダルの検索ダイナミッククエリの精製大型言語モデル (LLM)マルチモダルの表現学習意味論的推論

さらに関連する動画

05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

570

12:55

Multimodal Protocol for Assessing Metacognition and Self-Regulation in Adults with Learning Difficulties

Published on: September 27, 2020

8.5K

関連する概念動画

01:12

Retrieval

169

Retrieval is the process of getting information out of memory storage and back into conscious awareness. This ability is essential for daily tasks like brushing hair and teeth, driving to work, and performing job duties. Retrieval occurs in three ways: recall, recognition, and relearning.
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...

169

01:45

ER Retrieval Pathway

3.9K

In the secretory pathway, vesicles transport proteins from one cellular compartment to another in forward transport to deliver the protein to its correct location. Occasionally, misfolded proteins and incorrect proteins escape their original compartments, and a retrieval pathway is used to return the escaped proteins to their original compartment.
The ER uses many checkpoints to prevent the entry of incorrectly folded or a resident protein as cargo onto a transport vesicle. These mechanisms...

3.9K

01:15

Sensory Modalities

1.5K

Sensation typically is the process by which the sensory receptors and sense organs detect stimuli from the internal and external environment and transmit this information to the central nervous system for processing.
General senses refer to the broad category of sensory information detected by receptors in the body and can be further grouped into somatic and visceral senses. Somatic sensations include touch, pressure, temperature, and pain and are essential for navigating our environment and...

1.5K

01:25

The Anchoring-and-Adjustment Heuristic

7.4K

In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. However, sometimes, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the...

7.4K

01:07

Elaborative Rehearsals

129

Elaborative rehearsal is a crucial cognitive strategy that strengthens information encoding in long-term memory by making meaningful connections between new data and pre-existing knowledge. This approach contrasts with maintenance rehearsal, which involves simple repetition without delving into the significance of the information. While maintenance rehearsal might temporarily keep information active in short-term memory, it is less effective for long-term retention.
The effectiveness of...

129

01:22

Chunking and Rehearsal in Sensory Memory

293

Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...

293

関連する実験動画