Fencing

Fencing in CloudNativePG is the ultimate process of protecting thedata in one, more, or even all instances of a PostgreSQL cluster when theyappear to be malfunctioning. When an instance is fenced, the PostgreSQL serverprocess ( postmaster ) is guaranteed to be shut down, while the pod is kept running.This makes sure that, until the fence is lifted, data on the pod is not modified byPostgreSQL and that the file system can be investigated for debugging andtroubleshooting purposes.

How to fence instances

In CloudNativePG you can fence:

  • a specific instance

  • a list of instances

  • an entire Postgres Cluster

Fencing is controlled through the content of the cnpg.io/fencedInstances annotation, which expects a JSON formatted list of instance names.If the annotation is set to '["*"]' , a singleton list with a wildcard, thewhole cluster is fenced.If the annotation is set to an empty JSON list, the operator behaves as if theannotation was not set.

For example:

The annotation can be manually set on the Kubernetes object, for example viathe kubectl annotate command, or in a transparent way using the kubectl cnpg fencing on subcommand:

#  to fence only one instance
kubectl cnpg fencing on cluster-example 1

#  to fence all the instances in a Cluster
kubectl cnpg fencing on cluster-example "*"

Here is an example of a Cluster with an instance that was previously fenced:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
    annotations:
      cnpg.io/fencedInstances: ["cluster-example-1"]
[...]

How to lift fencing

Fencing can be lifted by clearing the annotation, or set it to a different value.

As for fencing, this can be done either manually with kubectl annotate , orusing the kubectl cnpg fencing subcommand as follows:

#  to lift the fencing only for one instance
#  N.B.: at the moment this wont work if the whole cluster was fenced previously,
#        in that case you will have to manually set the annotation as explained above
kubectl cnpg fencing off cluster-example 1

#  to lift the fencing for all the instances in a Cluster
kubectl cnpg fencing off cluster-example "*"

How fencing works

Once an instance is set for fencing, the procedure to shut down the postmaster process is initiated. This consists of an initial smart shutdownwith a timeout set to .spec.stopDelay , followed by a fast shutdown ifrequired. Then:

  • the Pod will be kept alive

  • the Pod won’t be marked as Ready

Warning

When at least one instance in a Cluster is fenced, failovers/switchovers for that Cluster will be blocked until the fence is lifted, as the status of the Cluster cannot be considered stable.

In particular, if a primaryinstance will be fenced, the postmaster process will be shut down but no failover will happen, interrupting the operativity of the applications. When the fence will be lifted, the primary instance will be started up again without any failover happening.

Given that, we advise the user to fence only replica instances when possible.

Warning

If the primary is the only fenced instance in a Cluster and the pod is deleted, a failover will be performed. When the fence on the old primary is lifted, that instance is restarted as a standby (follower of the new primary).

If a fenced instance is deleted, the pod will be recreated normally, but thepostmaster won’t be started. This can be extremely helpful when instancesare Crashlooping .