From Assistants to Agents: Evaluating Autonomous LLM Agents in Real-World DevOps Pipeline

Abstract

The addition of autonomous large language model (LLM) agents into the DevOps process is a paradigm shift in terms of software development and deployment automation. This paper assesses the performance of autonomous LLM agents in different stages of the DevOps process, such as development, testing, deployment, and monitoring. The objectives are to evaluate the operational effectiveness of LLM agents, their scalability, and the quality of their decision-making, and to compare the performance of the agents with that of conventional DevOps tools. The study uses case studies and an intensive data collection method to analyze such critical performance indicators as the deployment time, the error rates, and the efficiency of automation. The results indicate that LLM agents should be able to achieve a substantial reduction in human intervention, improve the speed of automation, and maximize decision-making in CI/CD pipelines. The main contributions to the paper are the evaluation framework of the LLM agents and suggestions to enhance their integration and application in practice.