On the variability of generative artificial intelligence methods in conceptual modeling: an experimental evaluation...

Large Language Models (LLMs) increasingly support time-consuming conceptual modeling tasks such as building or combining models from text. In this context, causal maps are labeled, directed graphs that represent causal relationships (edges) between selected factors (nodes) in a problem domain. Modelers control the output through the LLM's prompt and decoding parameters such as temperature, and recent studies either use a default temperature or minimize it by expecting that it supports replicability. However, evidence about this expectation remains limited, as does our understanding of how different LLMs and decoding or prompting strategies shape results. We assess the extent to which findings are an artifact of the LLM or its variability rather than a reflection of the modeler's controls. We focus on the common conceptual modeling task of combining models, exemplified through three algorithms that reflect a variety of prompting techniques. We use five LLMs (DeepSeek, Gemini, GPT, Qwen, Ministral) and different versions to account for the potential impact of the architectures. Using the Design of Experiments approach of factorial analyses (supplemented by paired tests and confidence interval checks), our results show that outcomes differ significantly across LLMs and versions. Depending on the algorithm, randomness from the LLM can be the dominating factor, responsible for up to 96% of the variation in results. This happens even if we directly control the decoding, as evidenced by our comparison of greedy, beam-search, and contrastive decoding. We find that prompt effects are systematic but metric-dependent, while decoding affects performance but there is no universally best strategy. Since our findings suggest that replicability of conceptual modeling studies using LLMs may be limited, future studies may report the distribution of performance measures alongside exact model, decoding, replication, and baseline-normalization settings.