Witness nodes
=============

A witness node is a lightweight node which functions as a data node but
which does not store or replicate data. It is used to allow a PGD
cluster which uses Raft consensus to have an odd number of voting nodes
and therefore be able to achieve a majority when making decisions.

Witness Nodes within PGD groups or regions
------------------------------------------

One typical use of witness nodes is when a PGD group has two data nodes
but resources are not available for the recommended three data nodes. In
this case, a witness node can be added to the PGD group to provide a
third voting node to local Raft decision making. These decisions are
primarily about who will be electing a write leader for the proxies to
use. With only two nodes, there could be no consensus over which data
node could be write leader. With two data nodes and a witness, there
would be two candidates (the data nodes) and three voters (the data
nodes and the witness) normally, and when a data node was down, there
would still be two voters able to select a write leader.

Witness Node outside regions
----------------------------

At a higher level, witness nodes can be used when there are multiple PGD
groups mapped to different regions. For example with three data nodes
per-region in two regions, while running normally allows all six data
nodes to participate in Raft decisions and obtaining DDL and DML global
locks. Even when a data node is down, there are sufficient data nodes to
obtain a consensus. But if a network partition occurs and connectivity
with the other region is lost, then there are now only three nodes out
of six available, which is not enough for a consensus. To avoid this
scenario, a witness node in a third region can be deployed as part of
the PGD cluster. This witness node will allow a consensus to be achieved
for most operational requirements of the PGD cluster while a region is
unavailable.