Problem

The increasing adoption of Large Language Models (LLMs) has begun to redefine software development, offering new capabilities for tasks like code generation and maintenance. However, individual LLMs still face significant limitations, including catastrophic forgetting, hallucinations, privacy concerns, and restricted reasoning for complex problems. To address these issues, LLM-based multi-agent systems (MAS) have emerged, proposing a collaborative approach where multiple specialized agents work together. Despite their promise, developers encounter challenges in selecting appropriate technologies, designing effective coordination rules, and defining agent roles within these systems. A critical gap exists in the systematic and detailed comparison of MAS frameworks, hindering both researchers and practitioners in understanding their capabilities, performance, and suitability for diverse software engineering tasks.

Method

This study employed a mixed-method approach, combining qualitative and quantitative analyses to evaluate existing MAS frameworks for software engineering. First, a qualitative analysis was conducted to identify foundational MAS concepts from academic literature and to select prominent open-source frameworks from GitHub based on popularity and documentation quality. These 16 frameworks were then evaluated against technical characteristics covering installation, developer interface, agent and tool integration, agent creation, orchestration, and monitoring capabilities. Based on this, 8 representative frameworks were chosen for further investigation. Second, a quantitative empirical evaluation was performed using the 4 most popular frameworks (AutoGen, AutoGPT, Dify, Semantic Kernel). The chosen use case was the summarization of GitHub README.MD files, implemented using a multi-agent pipeline called Metagente, which involves optimization and evaluation phases. The study utilized existing datasets (TS10, TS50 for training; ES for testing) and standardized experimental parameters, including LLM types (GPT-4o, GPT-4o-mini), temperature, maximum iterations, and a ROUGE-L threshold. Effectiveness was measured using ROUGE-1, ROUGE-2, and ROUGE-L scores, while efficiency was assessed by token usage, number of requests, and total execution time.

Results

The qualitative analysis revealed that existing MAS frameworks generally provide good coverage of fundamental components such as coordination rules, role specification, and message handling. However, advanced features like integrated benchmarking, monitoring, and human-in-the-loop capabilities are often still underdeveloped or reliant on external tools. Low-code frameworks tended to offer more built-in qualitative aspects, while high-code frameworks provided greater flexibility at the cost of increased development effort. In the quantitative evaluation, regarding effectiveness, no significant differences in summarization accuracy (ROUGE scores) were observed across the tested frameworks. Dify generally achieved the highest average ROUGE scores, closely followed by Semantic Kernel. Interestingly, increasing the training sample size for prompt optimization did not consistently improve performance for most frameworks. For efficiency, Semantic Kernel Chat exhibited substantially higher token usage and a greater number of API requests, primarily due to its internal orchestration mechanism relying heavily on the model for agent communication and decision-making. AutoGPT, on the other hand, demonstrated significantly longer execution times in both optimization and evaluation phases, attributed to its internal application implementation. The other frameworks showed more consistent and lower resource consumption.

Implications

This research offers a practical guide for developers navigating the selection of LLM-based MAS frameworks, highlighting the importance of aligning framework choice with specific project requirements and complexity. For practitioners, the findings underscore the trade-offs between flexibility, ease of use, and the level of control desired over agent orchestration. Researchers gain a foundational methodology for evaluating LLM-based MAS. Key lessons learned include the nascent state of monitoring capabilities in current frameworks, the critical role of robust agent communication protocols, and the urgent need for standardized empirical metrics to evaluate the performance of rapidly evolving MAS. Furthermore, the study confirms that VectorDBs are a common and effective solution for supporting long-term memory in MAS, often combined with short-term memory approaches. Identified challenges include the immaturity and incompleteness of many MAS tools, with some frameworks lacking essential functionalities or comprehensive documentation. Additionally, while low-code frameworks simplify development, they often lack the advanced features and flexibility required for more complex or custom MAS applications. Future work should expand evaluations to a broader range of software engineering tasks, incorporate diverse benchmarks, and investigate real-world industrial use cases to further assess framework effectiveness and generalizability.