Establishing human responsibility and accountability at early stages of the lifecycle for AI-based defence systems

The use of AI technologies in weapons systems has triggered a decade-long international debate, especially with regard to human control, responsibility, and accountability around autonomous and intelligent systems (AIS) in defence. However, most of these ethical and legal discussions have revolved primarily around the point of use of a hypothetical AIS, and in doing so, one critical component still remains under-appreciated: human decision-making across the full timeline of the AIS lifecycle. When discussions around human involvement start at the point at which a hypothetical AIS has taken some undesirable action, they typically prompt the question: "what happens next?" This approach primarily concerns the technology at the time of use and may be appropriate for conventional weapons systems, for which humans have clear lines of control and therefore accountability at the time of use. However, this is not precisely the case for AIS. Rather than focusing first on the system in its comparatively most autonomous state, it is more helpful to consider when, along the lifecycle, humans have more clear, direct control over the system (e.g. through research, design, testing, or procurement) and how, at those earlier times, human decision-makers can take steps to decrease the likelihood that an AIS will perform 'inappropriately' or take incorrect actions. In this paper, we therefore argue that addressing many arising concerns requires a shift in how and when participants of the international debate on AI in the military domain think about, talk about, and plan for human involvement across the full lifecycle of AIS in defence. This shift includes a willingness to hold human decision-makers accountable, even if their roles occurred at much earlier stages of the lifecycle. Of course, this raises another question: "How?" We close by formulating a number of recommendations, including the adoption of the IEEE-SA Lifecycle Framework, the consideration of policy knots, and the adoption of Human Readiness Levels.