In the age of digitization when milliseconds are equal to user happiness and downtime equates to millions lost, the hard currency of software development today is resilience.
This is the high-speed world in which Chukwubuikem Victory Onwukwe labours, not as a seasoned senior software engineer but as a systems architect whose creations are designed to weather the whimsical tempest of scale.
Friends and colleagues refer to Victory by such a designation because he has spent nearly a decade building the unheralded infrastructure necessary to keep systems up and running. His work does not shine. Nor is it in terms of cool UI or fashionable apps. Rather it’s in the hum of perfect distributed services, in the unobtrusive recovery of a dying cluster node, in the solid confidence that critical transactions will get through when the unexpected occurs.
From his early work as a single-threaded-application developer to designing large-scale distributed systems today, Victory’s life has been marked by a single passion: how to design systems to fail elegantly. His interest in fault tolerance was ignited by systems messiness in the real world instead of theoretical academic work.
Working at one of West Africa’s rapidly expanding fintech companies, he witnessed how a minor network problem could become an entire outage blocking users from accessing vital services.
The post-mortem that came next did not simply tell him how things had broken; it showed him a path to a different sort of engineering predicated on humility in the presence of complexity and a strict respect for failure.
Resilience to Victory is not a checkmark or a dependency on a library. It’s a mindset. Anticipating failure and the understanding that even the best system in a theoretical world has a certain behaviour when under pressure, amidst latency, hardware failure, cascading time-outs, or even simple human error as to what really defines its resilience.
At its core is a belief that fault tolerance is not a patch but a pattern to be interwoven into the architecture. His systems are constructed using lessons from a couple of decades of learning in distributed computing: graceful degradation, bulkheading, circuit breaking, idempotency, and eventual consistency. But Victory’s interpretation of these ideas goes deeper than case studies. The application layer is but the outer skin to a system’s resilience story, he believes. The real work is on how things talk to each other, how state is dealt with across flaky networks, how systems recover from failure, and how systems recover from partial failure without the aid of some sort of intervention.
One of Victory’s standout projects was the re-architecting of a transactional platform used by millions of users on a daily basis. The legacy system had a brittle core—coupled services spreading failure down the stack. Instead of advocating a full rewrite, Victory advocated an incremental technique with service boundaries and observability hooks with carefully crafted contracts. Redundancy was added to queues, distributed tracing was used to identify bottlenecks, and compensating transactions were used to deal with data consistency without adding locking or downtime. In a span of a year, the system was refactored from an unreliable monolith to a resilient service mesh capable of surviving node crashes, unexpected spikes in traffic, and even geographical outages.
And perhaps Victory’s most subtle realization is to recognize that resilience is a technical issue but also an org problem. Systems reflect the organizations that build them, and Victory invests a lot in mentoring junior engineers, in conducting post-incident reviews focused on learning over blame, and in building the type of operational excellence in which uptime is a shared concern. His words are as thoughtful as his code, architecting failure modes not yet seen but anticipated by history and by modeling out scenarios.
When asked about the strong architecture of the next century, Victory is generally reluctant to talk. He does not perceive the ascendance of server-less and edge computing as panaceas but as transformations accompanied by new types of failure modes.
He is skeptical of abstractions that distance engineers from the knowledge of systems over which they’re building. “Resilience,” he remarks, “comes from a knowledge of what you’re standing on.” In an era increasingly enthralled by automation and black-box services, his commitment to first-principle thought is a welcome respite.
As systems become larger and more interdependent and distributional in nature, the price of not taking long and careful thought about failure skyrockets.
In such a case, the work of engineers such as Chukwubuikem Victory Onwukwe aren’t just useful, they are indispensable. An unsung hero to system design circles, he’s a good role model to all of us on how the best systems are not systems that never fail but systems constructed by engineers who comprehend failure better than anyone and construct systems anyway.