CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows

📖 论文核心内容

1. 主要解决了什么问题？

The core problem addressed by the paper is the challenge of diagnosing and repairing Continuous Integration (CI) failures in modern software development workflows. Unlike traditional program repair, CI failures often involve non-code artifacts, environmental and dependency issues, noisy execution logs, and workflow-level constraints, which are not adequately captured by existing benchmarks. Current program repair benchmarks are test-centric, focus solely on source code, assume fixed execution environments, and rely on simplified CI workflows, leaving a significant research gap in repository-level validation under real-world CI conditions. This problem matters because CI workflows are central to ensuring software correctness and reliability, and improving automated repair mechanisms for CI failures can significantly enhance developer productivity and software quality.

2. 提出了什么解决方案？

The paper introduces CI-Repair-Bench, a repository-aware benchmark specifically designed for automated patch validation via CI workflows. The benchmark is constructed from real GitHub Actions executions and includes 567 CI failure instances from 103 repositories. It categorizes failures into 12 distinct CI error types, enabling fine-grained evaluation of repair strategies. The key innovation lies in evaluating repair correctness exclusively through full CI re-execution under original workflows, ensuring realism and accuracy. This approach differs from existing benchmarks by incorporating repository-level constraints, non-code artifacts, and real-world CI environments, providing a more comprehensive foundation for advancing research in CI-native automated program repair.

3. 核心方法/步骤/策略

The methodology involves constructing the CI-Repair-Bench benchmark using real-world data from GitHub Actions executions. CI failures are categorized into 12 error types, including formatting, linting, dependency issues, and configuration errors. The benchmark evaluates repair correctness through full CI re-execution under the original workflows, ensuring that repairs are validated in realistic conditions. A reference CI repair workflow is provided, which analyzes CI logs to localize faults and generate candidate patches. This workflow leverages tools and techniques for fault localization and patch generation, emphasizing the importance of repository-level context and workflow constraints. Implementation details include the use of large language models (LLMs) for automated repair and the integration of CI logs for error diagnosis.

4. 实验设计

The experiments are designed to evaluate the effectiveness of automated repair strategies using the CI-Repair-Bench benchmark. Metrics include repair success rates, categorized by CI error types, and comparisons across different repair approaches. Baselines include traditional program repair methods and large language models (LLMs). Empirical results show that automated repair is most effective for localized, tool-enforced failures such as formatting and linting, achieving higher success rates. However, environment, dependency, and configuration-related failures remain challenging, with the best-performing LLM achieving an overall repair success rate of 18.9%. These results highlight the strengths and limitations of current automated repair techniques in addressing diverse CI failure scenarios.

5. 结论

The paper concludes that CI-Repair-Bench provides a realistic and comprehensive evaluation foundation for advancing research on CI-native automated program repair. The benchmark's repository-aware design and error-type categorization enable fine-grained analysis of repair strategies, revealing that automated repair is effective for certain error types but struggles with complex failures involving dependencies and configurations. Limitations include the relatively low success rates for challenging error types and the reliance on existing tools and models. Future directions include improving repair techniques for complex CI failures, integrating more sophisticated fault localization methods, and expanding the benchmark to cover additional repositories and error scenarios. The study underscores the importance of repository-level context in automated program repair research.

🤔 用户关心的问题

• How does CI-Repair-Bench leverage large language models (LLMs) for automated patch generation, and what specific techniques are used to localize faults in CI failures? This question directly addresses the user's interest in how LLMs are utilized for generating patches and localizing bugs, which is a core aspect of the benchmark's reference repair workflow.
• What are the repair success rates of LLMs across the 12 categorized CI error types, and how do these rates vary for semantic, syntax, and configuration-related bugs? The user is interested in repair performance across different bug types, and this question probes into the empirical results provided in the paper to understand how LLMs perform across diverse error categories.
• How does CI-Repair-Bench validate the correctness of patches generated by LLMs, and what role does full CI re-execution play in ensuring reliability of repairs? Patch validation is a key interest for the user, and this question explores the benchmark's methodology for evaluating patch correctness, particularly the use of CI workflows for validation.
• What challenges do LLMs face when repairing environment, dependency, and configuration-related failures, and how might static or dynamic analysis techniques be integrated to improve repair reliability in these cases? This question aligns with the user's interest in improving repair reliability through interaction with static/dynamic analysis, focusing on the challenging error types identified in the paper.
• How does CI-Repair-Bench's repository-aware design and real-world CI workflow data contribute to advancing LLM-based program repair research, particularly in comparison to traditional test-centric benchmarks? The user is interested in how CI-native approaches differ from traditional ones, and this question probes into the benchmark's unique design and its implications for LLM-based repair research.

💡 逐项解答

📝 综合总结

CI-Repair-Bench leverages large language models (LLMs) as a core component of its automated patch generation process, particularly within its reference CI repair workflow. The paper describes how LLMs are employed to analyze CI logs and generate candidate patches, emphasizing their ability to interpret noisy execution logs and workflow-level constraints. Specifically, the benchmark uses LLMs to address localized, tool-enforced failures, such as formatting and linting issues, which are more amenable to automated repair. The empirical results highlight that the best-performing LLM achieves an 18.9% repair success rate, showcasing its potential but also underscoring the challenges of repairing more complex failures related to environment, dependency, and configuration issues.

Fault localization in CI failures is achieved through log analysis, which serves as the foundation for identifying the root cause of errors. The paper notes that the workflow "analyzes CI logs to localize faults," enabling targeted patch generation. This approach is particularly effective for failures with clear, tool-enforced constraints, as these provide structured feedback that can be directly mapped to corrective actions. However, the benchmark also acknowledges the limitations of this method when dealing with failures arising from less deterministic factors, such as environmental inconsistencies or dependency conflicts, which require deeper contextual understanding and broader repository-level insights.

The integration of LLMs into CI-Repair-Bench reflects a significant advancement in automated program repair, as it moves beyond traditional test-centric benchmarks to address the complexities of CI workflows. By focusing on repository-aware validation and leveraging LLMs for both fault localization and patch generation, the benchmark provides a realistic evaluation framework that aligns with the challenges faced in modern software development. This dual focus on localization and repair highlights the importance of combining log analysis with the generative capabilities of LLMs to tackle diverse CI failure scenarios effectively.

The paper provides detailed insights into the repair success rates of large language models (LLMs) across 12 categorized CI error types, emphasizing the challenges posed by semantic, syntax, and configuration-related bugs. Empirical results reveal that LLMs perform best on localized, tool-enforced failures such as formatting and linting errors, achieving relatively higher success rates in these categories. Specifically, the authors note that "automated repair is most effective for localized, tool-enforced failures," which suggests that these errors are easier to diagnose and resolve due to their deterministic nature and clear feedback from CI tools.

However, the repair success rates drop significantly for more complex error types, particularly those involving semantic, environment, dependency, and configuration-related issues. The paper highlights that "environment, dependency, and configuration-related failures remain challenging," with the best-performing LLM achieving only an 18.9% overall repair success rate. This disparity underscores the inherent difficulty in addressing bugs that require deeper contextual understanding or involve external factors such as system configurations and dependencies. Semantic bugs, which often require nuanced comprehension of code logic and intent, similarly pose significant challenges, although specific success rates for semantic errors are not explicitly broken down in the paper.

The variation in repair performance across error types reflects the limitations of current LLMs in handling non-code artifacts and workflow-level constraints, which are central to CI failures. These findings highlight the need for further advancements in LLM capabilities to better address the complexities of semantic and configuration-related bugs, as well as the importance of benchmarks like CI-Repair-Bench in driving research toward more effective CI-native automated repair solutions.

CI-Repair-Bench employs a rigorous methodology to validate the correctness of patches generated by large language models (LLMs), emphasizing the importance of full CI re-execution within original workflows. According to the paper, the benchmark evaluates repair correctness exclusively through "full CI re-execution under original workflows," ensuring that patches are tested in the same environment and conditions as the original failure. This approach is critical because CI failures often involve complex interactions between code, dependencies, and environment configurations, which cannot be reliably assessed through isolated or simplified testing methods.

The role of full CI re-execution is pivotal in ensuring the reliability of repairs. By re-running the entire CI workflow, the benchmark verifies whether the patch resolves the failure without introducing new issues or violating repository-level constraints. The paper highlights that CI failures are not limited to source code errors but can also stem from "non-code artifacts, environment and dependency issues, noisy execution logs, and workflow-level constraints." This complexity necessitates a validation process that mirrors the real-world CI environment, as traditional test-centric benchmarks fail to account for these factors.

Furthermore, the benchmark categorizes failures into 12 distinct CI error types, enabling fine-grained evaluation of patch effectiveness across different scenarios. Empirical results demonstrate that automated repair is most successful for "localized, tool-enforced failures such as formatting and linting," where the scope of the issue is narrowly defined. However, more challenging failures related to environment, dependency, and configuration issues highlight the limitations of current automated repair techniques and underscore the need for comprehensive validation through CI workflows.

In summary, CI-Repair-Bench’s reliance on full CI re-execution ensures that patches are evaluated in a realistic and repository-aware context, addressing the multifaceted nature of CI failures. This methodology not only validates patch correctness but also provides insights into the strengths and weaknesses of automated repair approaches, paving the way for advancements in CI-native program repair research.

The paper highlights significant challenges faced by large language models (LLMs) when addressing environment, dependency, and configuration-related failures in Continuous Integration (CI) workflows. These types of failures are inherently complex because they often involve non-code artifacts, noisy execution logs, and dynamic, repository-specific constraints. According to the authors, "environment, dependency, and configuration-related failures remain challenging; the best-performing LLM achieves an 18.9% repair success rate." This low success rate underscores the difficulty of diagnosing and repairing such issues, which require understanding intricate interactions between software components, external dependencies, and the execution environment.

One of the key obstacles is the lack of structured information in CI logs, which are often verbose and noisy. The paper notes that existing repair workflows rely on log analysis to localize faults, but this approach struggles with failures that span multiple layers of the CI pipeline. For example, dependency-related issues may involve mismatched library versions or missing packages, while configuration errors could stem from incorrect settings in YAML files or environment variables. These problems are compounded by the dynamic nature of CI workflows, where the execution environment can vary significantly across repositories.

To improve repair reliability, the integration of static and dynamic analysis techniques could be transformative. Static analysis could help identify potential configuration and dependency issues before execution by analyzing repository metadata, configuration files, and dependency manifests. For instance, tools could statically verify compatibility between declared dependencies and the target environment. Dynamic analysis, on the other hand, could complement this by monitoring runtime behavior during CI execution to detect anomalies or mismatches in real-time. The paper’s emphasis on "repository-level validation" suggests that combining these approaches could enable more precise fault localization and patch generation tailored to the specific repository context.

In summary, while LLMs show promise in automating CI repair, their effectiveness is limited by the complexity of environment and configuration-related failures. Integrating static and dynamic analysis techniques offers a pathway to address these challenges by enhancing fault detection and repair precision, ultimately advancing the reliability of automated CI workflows.

CI-Repair-Bench’s repository-aware design and reliance on real-world CI workflow data mark a significant departure from traditional test-centric benchmarks, offering unique advantages for advancing LLM-based program repair research. Unlike conventional benchmarks that focus narrowly on source code and assume fixed execution environments, CI-Repair-Bench integrates the complexities of modern Continuous Integration (CI) systems. The paper highlights that CI failures often involve "non-code artifacts, environment and dependency issues, noisy execution logs, and workflow-level constraints," which traditional benchmarks fail to capture. By leveraging real GitHub Actions executions from 103 repositories and categorizing 567 CI failure instances into 12 error types, CI-Repair-Bench provides a granular, error-type-aware evaluation framework. This enables researchers to address challenges that are intrinsic to CI workflows, such as configuration and dependency management, which are typically overlooked in test-centric approaches.

The benchmark’s reliance on full CI re-execution under original workflows is particularly impactful for LLM-based repair systems. This design ensures that repair correctness is evaluated in a realistic, repository-level context rather than under simplified or artificial conditions. The paper notes that automated repair approaches demonstrated varying effectiveness across error types, with tool-enforced failures like formatting and linting being more amenable to repair, while environment and configuration-related issues proved more challenging. For instance, the best-performing LLM achieved only an 18.9% repair success rate, underscoring the difficulty of addressing CI-native errors. This empirical evidence highlights the need for benchmarks like CI-Repair-Bench that reflect the real-world complexities of CI systems, thereby pushing LLM-based research to develop more robust and context-aware repair strategies.

In comparison to traditional benchmarks, CI-Repair-Bench’s repository-aware design not only broadens the scope of repair evaluation but also aligns it more closely with practical software development scenarios. By incorporating real-world CI data and emphasizing repository-level validation, it provides a foundation for developing repair techniques that are both technically sophisticated and operationally relevant. This shift from a test-centric to a CI-native paradigm represents a critical step forward in the evolution of automated program repair research, particularly in the era of LLMs.