Volume 18 No 12 (2020)
Download PDF
Resilience Engineering in DevOps: Fault Injection and Chaos Testing for Distributed Systems
Yogesh Ramaswamy
Abstract
As modern software systems increasingly adopt distributed and cloud-native architectures, resilience has become a critical non-functional requirement. These systems operate in dynamic, failure-prone environments where network delays, hardware crashes, and service outages are common. Traditional reactive approaches to reliability, such as post-incident forensics or static redundancy models, are no longer sufficient. To meet the demands of always-on digital services, organizations are embracing resilience engineering—a discipline focused on designing systems that gracefully withstand and recover from unexpected disruptions.
In this context, fault injection and chaos testing have emerged as proactive techniques to validate system behavior under controlled failure conditions. Unlike conventional testing, which assumes ideal operational states, these methods deliberately introduce faults—such as service crashes, network partitions, and latency spikes—to observe how systems degrade and recover. This article explores the integration of fault injection practices into DevOps pipelines, enabling continuous validation of system resilience during the software delivery lifecycle.
Key contributions include a reference DevOps-resilience framework, integration strategies using tools like Chaos Monkey, Gremlin, and LitmusChaos, and techniques for defining blast radius, steady-state metrics, and rollback automation. The article demonstrates how chaos testing can be embedded into CI/CD workflows via Kubernetes-native operators and GitOps policies, ensuring repeatability and safety.
The proposed methods were evaluated using a microservices-based e-commerce application deployed on a Kubernetes cluster. Through systematic fault experiments—such as pod termination, node failure, and artificial latency—metrics such as Mean Time to Detect (MTTD), Mean Time to Recovery (MTTR), and error budget consumption were measured. Results show that systems exposed to ongoing resilience validation recovered 32% faster and experienced 45% fewer unplanned outages compared to systems without such testing. These findings underscore the practical value of embedding chaos engineering into DevOps for robust, self-healing systems.
Keywords
: Software systems, distributed and cloud-native architectures, and non-functional requirement
Copyright
Copyright © Neuroquantology
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Articles published in the Neuroquantology are available under Creative Commons Attribution Non-Commercial No Derivatives Licence (CC BY-NC-ND 4.0). Authors retain copyright in their work and grant IJECSE right of first publication under CC BY-NC-ND 4.0. Users have the right to read, download, copy, distribute, print, search, or link to the full texts of articles in this journal, and to use them for any other lawful purpose.