A systematic mapping study method for software architecture research with LLM-assisted quality assessment

This article describes the protocol used to conduct a systematic mapping study (SMS) of software architecture. The presence of architectural and design instruction in primary studies cannot be determined through search and superficial review alone. A principled quality assessment must be conducted to identify studies that are both relevant and of high quality. LLM assistance offers practical means of performing such assessments in a timely and efficient manner. Therefore, our contribution is threefold: • Guidance for the following SMS steps: tooling setup, database search, deduplication and snowballing, primary study selection, quality assessment, and data extraction. • The rationale and methodology to perform quality assessment on primary studies regarding software architecture. • A method of using large language models (LLMs) together with human evaluators is presented. The method is supported by bespoke open-source software.