Keepalived, HAProxy, and the Firewall That Ate My VRRP Packets

High-Availability Gone Split-Brain: A Production Debugging Story

1. Introduction

High availability is one of those things you configure once and expect to fade into the background — until it doesn’t. In our environment, HAProxy load-balances backend traffic, and Keepalived provides the failover layer via the Virtual Router Redundancy Protocol (VRRP). A virtual IP (VIP) floats between two or more nodes; if the active node fails, a backup takes over within seconds.

VIPs claims can effect the applicaiton calls

Recently, that calm surface cracked. A routine check showed that every node in a VRRP cluster had claimed the VIPs. This is the classic split-brain scenario, and it can lead to routing loops, duplicate ARP responses, and unpredictable application behavior.

This article walks through the investigation, the two distinct root causes we found, and the security lessons we took away about VRRP and host-based firewalls.

2. Architecture Overview

Our stack looks like this:

┌─────────────────┐
     Client ───────►│  Virtual IP     │
     requests       │  10.10.3.9/10 │
                    └────────┬────────┘
                             │
            ┌────────────────┼────────────────┐
            │…