Executive-centered AI? Designing predictive systems for the public sector

Area of Science:

• Sociology of Technology
• Human-Computer Interaction
• Artificial Intelligence Ethics

Background:

• Recent discourse advocates for artificial intelligence (AI) to evolve into intelligence augmentation (IA), focusing on human enhancement.
• This shift is promoted through policies and research, aiming for AI systems that center on and amplify human capabilities.

Purpose of the Study:

• To investigate the practical enactment of AI development within an AI company.
• To analyze how two predictive systems were developed for public sector accounting and healthcare stakeholders.
• To examine the embedded values, objectives, and labor divisions within these AI systems, particularly concerning their impact on professionals.

Main Methods:

• Field study conducted at an AI company.
• Analysis of the development process of two predictive AI systems.
• Application of Science and Technology Studies (STS) theories, focusing on values in design.
• Empirical data analysis of objectives, structured performances, and divisions of labor.

Main Results:

• AI system development was influenced by managerial interests prioritizing cost-efficiency.
• The AI systems were designed as managerial tools for efficiency improvements and cost reductions.
• These systems were implemented in a top-down manner, imposed on professionals in accounting and healthcare sectors.

Conclusions:

• The transformation of AI into IA appears questionable in practice.
• The study raises critical questions about the true meaning of human-centered AI and its practical realization.
• Further research is needed to redefine human-machine relationships in the era of big data and AI for genuine ethical and responsible AI development.