[TIL] Survey about Chubby/Zookeeper and their fault tolerance

July 25th, 2016

Difference between Chubby and Zookeeper

Consensus Algorithm:
- Chubby: Using Paxos consensus algorithm
- Zookeeper: Using ZAB (which is a modified algorithm of Paxos)
Access Path:
- Chubby: Must through leader, follower don’t accept any command directly.
- Zookeeper: Can accept any command from all follower, but will go back to leader.
Data Out-Of-Date:
- Chubby: No rish on out-of-date, because all data fro Leader.
- Zookeeper: Will get out-of-date data, if read without sync command. Sync command will force ask follower ask Leader first before return result to client.

The tolerance of Quorum-Backup and Primary-Backup:

Majority Quorums:

For Paxos or ZAB, A leader election need meet majority quorum ( n/2 +1 ) . It means if n=2f at most the service can tolerance f service failed at the same time. ( Quorum-Backup Replica ).

Primary-Backup

All data must confirm by all follower confirmed.
Much slower than “Quorum-Backup”.

Kafka ISR (In-Sync Replica)

In-Sync Replica is a backup policy which similar with Primary-Backup but in sychronous replication.

Server store two copy of replica data:

ISR (The latest one) store in Leader.
Follower store replica data might later than ISR.
Use Zookeeper to update all follower replica data from ISR.

In this case, if total server is n=f+1, we could tolerance number of f service failed. ( Primary-Backup replica )

trade-off:

Follower data critical late than leader, but in Kafka this case might be acceptable.
If follower cannot catch up data from leader, will drop ISR by leader.
If failed on leader, it has high possibility lost data if next leader. (Choose leader from non-ISR list, because all follower in ISR leave behind.)
Use Zookeeper to store ISR will introduce split brain issue. (two majority group issue).

Refer to Kafka ISR