Intent-driven LLM ensemble planning for flexible multi-robot manipulation

In this study, we address intent-driven task planning for complex multi-action manipulation sequences in heterogeneous multi-robot cells. Given a perception back-end that outputs a structured object-level scene description and a human operator's natural-language intent, we generate a precedence-consistent object-level robot-action sequence, which can then be executed by passing each such action to a lower-level motion planning module.

The pipeline integrates i. perception-to-text scene encoding, ii. an ensemble of large language models (LLMs) that generate candidate action sequences based on the operator's intent, iii. an LLM-based verifier that enforces formatting and precedence constraints, and iv. a deterministic consistency filter that rejects hallucinated objects. The pipeline is evaluated on an example task in which two robot arms work collaboratively to dismantle an electric-vehicle (EV) battery for recycling applications. A variety of components must be grasped and removed in specific sequences, determined either by human instructions or by task-order feasibility decisions made by the autonomous system.

On 200 real scenes with 600 operator prompts across five component classes, we used metrics of full-sequence correctness and next-task correctness to evaluate and compare five LLM-based planners (including ablation analyses of pipeline components). We also evaluated the LLM-based human interface in terms of time to execution and NASA TLX using human participant experiments. On 200 real scenes and 600 prompts, full-sequence correctness improves from 0.761 (single LLM) to 0.824 (6-LLM + verifier + deterministic filter), and next-object correctness improves from 0.866 to 0.894.

Results in our case study indicate that our ensemble-with-verification approach reliably maps operator intent to safe multi-robot plans while maintaining low user effort.