Problem

Large-scale open-source projects face challenges in managing hundreds of daily pull requests, each potentially introducing regressions. Existing methods for defect prediction often lack the precision needed for practical application in continuous integration workflows.

Method

DRS-OSS employs a fine-tuned Llama 3.1 8B sequence classifier trained on the ApacheJIT dataset. The model uses long-context representations combining commit messages, structured diffs, and change metrics. Techniques like 4-bit QLoRA and DeepSpeed ZeRO-3 CPU offloading enable efficient training on a single 20 GB GPU.

Results

DRS-OSS achieves state-of-the-art performance on the ApacheJIT benchmark with an F1 score of 0.64 and ROC-AUC of 0.89. Simulations show that gating only the riskiest 30% of commits can prevent up to 86.4% of defect-inducing changes.

Implications

DRS-OSS provides a deployable, production-oriented DRS pipeline with a FastAPI gateway, React-based dashboard, and GitHub App. This system delivers real-time risk feedback and is released with a full replication package.