Backup and Recovery¶

The operator can orchestrate a continuous backup infrastructurethat is based on the BarmanObjectStoreConfiguration tool. Insteadof using the classical architecture with a Barman server, whichbacks up many PostgreSQL instances, the operator relies on the barman-cloud-wal-archive , barman-cloud-check-wal-archive , barman-cloud-backup , barman-cloud-backup-list , and barman-cloud-backup-delete tools. As a result, base backups will be tarballs. Both base backups and WAL files can be compressed and encrypted.

For this, it is required an image with barman-cli-cloud installed.You can use the image ghcr.io/cloudnative-pg/postgresql for this scope,as it is composed of a community PostgreSQL image and the latest barman-cli-cloud package.

Important

Always ensure that you are running the latest version of the operands in your system to take advantage of the improvements introduced in Barman cloud (as well as improve the security aspects of your cluster).

A backup is performed from a primary or a designated primary instance in a Cluster (please refer to Replica clusters for more information about designated primary instances).

Cloud provider support¶

You can archive the backup files in any service that is supportedby the Barman Cloud infrastructure. That is:

You can also use any compatible implementation of thesupported services.

The required setup depends on the chosen storage provider and isdiscussed in the following sections.

S3¶

You will need the following information about your environment:

ACCESS_SECRET_KEY : the secret part of the previous access key
ACCESS_SESSION_TOKEN : the optional session token in case it is required

The access key used must have permission to upload files inthe bucket. Given that, you must create a k8s secret with thecredentials, and you can do that with the following command:

kubectl create secret generic aws-creds \
  --from-literal=ACCESS_KEY_ID=<access key here> \
  --from-literal=ACCESS_SECRET_KEY=<secret key here>
#  --from-literal=ACCESS_SESSION_TOKEN=<session token here> # if required

The credentials will be stored inside Kubernetes and will be encryptedif encryption at rest is configured in your installation.

Given that secret, you can configure your cluster like inthe following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "<destination path here>"
      s3Credentials:
        accessKeyId:
          name: aws-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: aws-creds
          key: ACCESS_SECRET_KEY

The destination path can be every URL pointing to a folder wherethe instance can upload the WAL files, e.g. s3://BUCKET_NAME/path/to/folder .

Other S3-compatible Object Storages providers¶

In case you’re using S3-compatible object storage, like MinIO orLinode Object Storage, you can specify an endpoint instead of using thedefault S3 one.

In this example, it will use the bucket bucket of Linode in the region us-east1 .

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "<destination path here>"
      endpointURL: bucket.us-east1.linodeobjects.com
      s3Credentials:
        [...]

Important

Suppose you configure an Object Storage provider which uses a certificate signed with a private CA, like when using MinIO via HTTPS. In that case, you need to set the option endpointCA referring to a secret containing the CA bundle so that Barman can verify the certificate correctly.

Note

If you want ConfigMaps and Secrets to be automatically reloaded by instances, you can add a label with key cnpg.io/reload to it, otherwise you will have to reload the instances using the kubectl cnpg reload subcommand.

MinIO Gateway¶

Optionally, you can use MinIO Gateway as a common interface whichrelays backup objects to other cloud storage solutions, like S3 or GCS.For more information, please refer to MinIO official documentation .

Specifically, the CloudNativePG cluster can directly point to a localMinIO Gateway as an endpoint, using previously created credentials and service.

MinIO secrets will be used by both the PostgreSQL cluster and the MinIO instance.Therefore you must create them in the same namespace:

kubectl create secret generic minio-creds \
  --from-literal=MINIO_ACCESS_KEY=<minio access key here> \
  --from-literal=MINIO_SECRET_KEY=<minio secret key here>

Note

Cloud Object Storage credentials will be used only by MinIO Gateway in this case.

Important

In order to allow PostgreSQL to reach MinIO Gateway, it is necessary to create a ClusterIP service on port 9000 bound to the MinIO Gateway instance.

For example:

apiVersion: v1
kind: Service
metadata:
  name: minio-gateway-service
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Warning

At the time of writing this documentation, the official MinIO Operator for Kubernetes does not support the gateway feature. As such, we will use a deployment instead.

The MinIO deployment will use cloud storage credentials to upload objects to theremote bucket and relay backup files to different locations.

Here is an example using AWS S3 as Cloud Object Storage:

apiVersion: apps/v1
kind: Deployment
[...]
spec:
  containers:
  - name: minio
    image: minio/minio:RELEASE.2020-06-03T22-13-49Z
    args:
    - gateway
    - s3
    env:
    # MinIO access key and secret key
    - name: MINIO_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: minio-creds
          key: MINIO_ACCESS_KEY
    - name: MINIO_SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: minio-creds
          key: MINIO_SECRET_KEY
    # AWS credentials
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: aws-creds
          key: ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: aws-creds
          key: ACCESS_SECRET_KEY
#  Uncomment the below section if session token is required
#    - name: AWS_SESSION_TOKEN
#      valueFrom:
#        secretKeyRef:
#          name: aws-creds
#          key: ACCESS_SESSION_TOKEN
        ports:
        - containerPort: 9000

Proceed by configuring MinIO Gateway service as the endpointURL in the Cluster definition, then choose a bucket name to replace BUCKET_NAME :

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: s3://BUCKET_NAME/
      endpointURL: http://minio-gateway-service:9000
      s3Credentials:
        accessKeyId:
          name: minio-creds
          key: MINIO_ACCESS_KEY
        secretAccessKey:
          name: minio-creds
          key: MINIO_SECRET_KEY
    [...]

Verify on s3://BUCKET_NAME/ the presence of archived WAL files beforeproceeding with a backup.

Azure Blob Storage¶

In order to access your storage account, you will need one of the following combinationsof credentials:

**Connection String**
Storageaccountname and **Storage account access key**
Storageaccountname and **Storage account SAS Token** .

The credentials need to be stored inside a Kubernetes Secret, adding data entries only whenneeded. The following command performs that:

kubectl create secret generic azure-creds \
  --from-literal=AZURE_STORAGE_ACCOUNT=<storage account name> \
  --from-literal=AZURE_STORAGE_KEY=<storage account key> \
  --from-literal=AZURE_STORAGE_SAS_TOKEN=<SAS token> \
  --from-literal=AZURE_STORAGE_CONNECTION_STRING=<connection string>

The credentials will be encrypted at rest, if this feature is enabled in the usedKubernetes cluster.

Given the previous secret, the provided credentials can be injected inside the clusterconfiguration:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "<destination path here>"
      azureCredentials:
        connectionString:
          name: azure-creds
          key: AZURE_CONNECTION_STRING
        storageAccount:
          name: azure-creds
          key: AZURE_STORAGE_ACCOUNT
        storageKey:
          name: azure-creds
          key: AZURE_STORAGE_KEY
        storageSasToken:
          name: azure-creds
          key: AZURE_STORAGE_SAS_TOKEN

When using the Azure Blob Storage, the destinationPath fulfills the followingstructure:

<http|https>://<account-name>.<service-name>.core.windows.net/<resource-path>

where <resource-path> is <container>/<blob> . The accountname ,which is also called storageaccountname , is included in the used host name.

Other Azure Blob Storage compatible providers¶

If you are using a different implementation of the Azure Blob Storage APIs,the destinationPath will have the following structure:

<http|https>://<local-machine-address>:<port>/<account-name>/<resource-path>

In that case, <account-name> is the first component of the path.

This is required if you are testing the Azure support via the Azure StorageEmulator or Azurite .

Google Cloud Storage¶

Currently, the operator supports two authentication methods for Google Cloud Storage,one assumes the pod is running inside a Google Kubernetes Engine cluster, the other one leveragesthe environment variable GOOGLE_APPLICATION_CREDENTIALS .

Running inside Google Kubernetes Engine¶

This could be one of the easiest way to create a backup, and only requiresthe following configuration:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "gs://<destination path here>"
      googleCredentials:
        gkeEnvironment: true

This, will tell the operator that the cluster is running inside a Google KubernetesEngine meaning that no credentials are needed to upload the files

Important

This method will require carefully defined permissions for cluster and pods, which have to be defined by a cluster administrator.

Using authentication¶

Following the instruction from Google you will get a JSON file that contains all the required information to authenticate.

The content of the JSON file must be provided using a Secret that can be createdwith the following command:

kubectl create secret generic backup-creds --from-file=gcsCredentials=gcs_credentials_file.json

This will create the Secret with the name backup-creds to be used in the yaml file like this:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "gs://<destination path here>"
      googleCredentials:
        applicationCredentials:
          name: backup-creds
          key: gcsCredentials

Now the operator will use the credentials to authenticate against Google Cloud Storage.

Important

This way of authentication will create a JSON file inside the container with all the needed information to access your Google Cloud Storage bucket, meaning that if someone gets access to the pod will also have write permissions to the bucket.

On-demand backups¶

To request a new backup, you need to create a new Backup resourcelike the following one:

apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: backup-example
spec:
  cluster:
    name: pg-backup

The operator will start to orchestrate the cluster to take therequired backup using barman-cloud-backup . You can checkthe backup status using the plain kubectl describe backup <name> command:

Name:         backup-example
Namespace:    default
Labels:       <none>
Annotations:  API Version:  postgresql.cnpg.io/v1
Kind:         Backup
Metadata:
  Creation Timestamp:  2020-10-26T13:57:40Z
  Self Link:         /apis/postgresql.cnpg.io/v1/namespaces/default/backups/backup-example
  UID:               ad5f855c-2ffd-454a-a157-900d5f1f6584
Spec:
  Cluster:
    Name:  pg-backup
Status:
  Phase:       running
  Started At:  2020-10-26T13:57:40Z
Events:        <none>

When the backup has been completed, the phase will be completed like in the following example:

Name:         backup-example
Namespace:    default
Labels:       <none>
Annotations:  API Version:  postgresql.cnpg.io/v1
Kind:         Backup
Metadata:
  Creation Timestamp:  2020-10-26T13:57:40Z
  Self Link:         /apis/postgresql.cnpg.io/v1/namespaces/default/backups/backup-example
  UID:               ad5f855c-2ffd-454a-a157-900d5f1f6584
Spec:
  Cluster:
    Name:  pg-backup
Status:
  Backup Id:         20201026T135740
  Destination Path:  s3://backups/
  Endpoint URL:      http://minio:9000
  Phase:             completed
  s3Credentials:
    Access Key Id:
      Key:   ACCESS_KEY_ID
      Name:  minio
    Secret Access Key:
      Key:      ACCESS_SECRET_KEY
      Name:     minio
  Server Name:  pg-backup
  Started At:   2020-10-26T13:57:40Z
  Stopped At:   2020-10-26T13:57:44Z
Events:         <none>

!!!Important This feature will not backup the secrets for the superuser and the application user. The secrets are supposed to be backed up as part of the standard backup procedures for the Kubernetes cluster.

Scheduled backups¶

You can also schedule your backups periodically by creating aresource named ScheduledBackup . The latter is similar to a Backup but with an added field, called schedule .

This field is a cron schedule specification, which follows the same: format used in Kubernetes CronJobs .

This is an example of a scheduled backup:

apiVersion: postgresql.cnpg.io/v1
kind: ScheduledBackup
metadata:
  name: backup-example
spec:
  schedule: "0 0 0 * * *"
  backupOwnerReference: self
  cluster:
    name: pg-backup

The above example will schedule a backup every day at midnight.

Hint

Backup frequency might impact your recovery time object (RTO) after a disaster which requires a full or Point-In-Time recovery operation. Our advice is that you regularly test your backups by recovering them, and then measuring the time it takes to recover from scratch so that you can refine your RTO predictability. Recovery time is influenced by the size of the base backup and the amount of WAL files that need to fetched from the archive and replayed during recovery (remember that WAL archiving is what enables continuous backup in PostgreSQL!). Based on our experience, a weekly base backup is more than enough for most cases - while it is extremely rare to schedule backups more frequently than once a day.

ScheduledBackups can be suspended if needed by setting .spec.suspend: true ,this will stop any new backup to be scheduled as long as the option is set to false.

In case you want to issue a backup as soon as the ScheduledBackup resource is createdyou can set .spec.immediate: true .

Note

.spec.backupOwnerReference indicates which ownerReference should be put inside the created backup resources.

none: no owner reference for created backup objects (same behavior as before the field was introduced) - self: sets the Scheduled backup object as owner of the backup - cluster: set the cluster as owner of the backup

WAL archiving¶

WAL archiving is enabled as soon as you choose a destination pathand you configure your cloud credentials.

If required, you can choose to compress WAL files as soon as theyare uploaded and/or encrypt them:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      [...]
      wal:
        compression: gzip
        encryption: AES256

You can configure the encryption directly in your bucket, and the operatorwill use it unless you override it in the cluster configuration.

PostgreSQL implements a sequential archiving scheme, where the archive_command will be executed sequentially for every WALsegment to be archived.

Important

By default, CloudNativePG sets archive_timeout to 5min , ensuring that WAL files, even in case of low workloads, are closed and archived at least every 5 minutes, providing a deterministic time-based value for your Recovery Point Objective (RPO). Even though you change the value of the archive_timeout , our experience suggests that the default value set by the operator is suitable for most use cases.

When the bandwidth between the PostgreSQL instance and the objectstore allows archiving more than one WAL file in parallel, youcan use the parallel WAL archiving feature of the instance managerlike in the following example:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      [...]
      wal:
        compression: gzip
        maxParallel: 8
        encryption: AES256

In the previous example, the instance manager optimizes the WALarchiving process by archiving in parallel at most eight readyWALs, including the one requested by PostgreSQL.

When PostgreSQL will request the archiving of a WAL that hasalready been archived by the instance manager as an optimization,that archival request will be just dismissed with a positive status.

Recovery¶

Cluster restores are not performed “in-place” on an existing cluster.You can use the data uploaded to the object storage to bootstrap anew cluster from a previously taken backup.The operator will orchestrate the recovery process using the barman-cloud-restore tool (for the base backup) and the barman-cloud-wal-restore tool (for WAL files, including parallel support, ifrequested).

For details and instructions on the recovery bootstrap method, please referto the Bootstrap from a backup ( `recovery )<Bootstrap from a backup ( recovery )>` .

Under the hood, the operator will inject an init container in the firstinstance of the new cluster, and the init container will start recovering thebackup from the object storage.

Important

The duration of the base backup copy in the new PVC depends on the size of the backup, as well as the speed of both the network and the storage.

When the base backup recovery process is completed, the operator starts thePostgres instance in recovery mode: in this phase, PostgreSQL is up, albeit notbeing able to accept connections, and the pod is healthy, according to theliveness probe. Through the restore_command , PostgreSQL starts fetching WALfiles from the archive (you can speed up this phase by setting the maxParallel option and enable the parallel WAL restore capability).

This phase terminates when PostgreSQL reaches the target (either the end of theWAL or the required target in case of Point-In-Time-Recovery). Indeed, you canoptionally specify a recoveryTarget to perform a point in time recovery. Ifleft unspecified, the recovery will continue up to the latest available WAL onthe default target timeline ( current for PostgreSQL up to 11, latest forversion 12 and above).

Once the recovery is complete, the operator will set the requiredsuperuser password into the instance. The new primary instance will startas usual, and the remaining instances will join the cluster as replicas.

The process is transparent for the user and it is managed by the instancemanager running in the Pods.

Retention policies¶

CloudNativePG can manage the automated deletion of backup files fromthe backup object store, using retentionpolicies based on recovery window.

Internally, the retention policy feature uses barman-cloud-backup-delete with --retention-policy “RECOVERY WINDOW OF {{ retention policy value }} {{ retention policy unit }}” .

For example, you can define your backups with a retention policy of 30 days asfollows:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      destinationPath: "<destination path here>"
      s3Credentials:
        accessKeyId:
          name: aws-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: aws-creds
          key: ACCESS_SECRET_KEY
    retentionPolicy: "30d"

Compression algorithms¶

CloudNativePG by default archives backups and WAL files in anuncompressed fashion. However, it also supports the following compressionalgorithms via barman-cloud-backup (for backups) and barman-cloud-wal-archive (for WAL files):

bzip2
gzip
snappy

The compression settings for backups and WALs are independent. See the: DataBackupConfiguration and WalBackupConfiguration sections inthe API reference.

It is important to note that archival time, restore time, and size changebetween the algorithms, so the compression algorithm should be chosen accordingto your use case.

The Barman team has performed an evaluation of the performance of the supportedalgorithms for Barman Cloud. The following table summarizes a scenario where abackup is taken on a local MinIO deployment. The Barman GitHub project includesa deeper analysis .

Compression	Backup Time (ms)	Restore Time (ms)	Uncompressed size (MB)	Compressed size (MB)	Approx ratio
None	10927	7553	395	395	1:1
bzip2	25404	13886	395	67	5.9:1
gzip	116281	3077	395	91	4.3:1
snappy	8134	8341	395	166	2.4:1

Tagging of backup objects¶

Barman 2.18 introduces support for tagging backup resources when saving them inobject stores via barman-cloud-backup and barman-cloud-wal-archive . As aresult, if your PostgreSQL container image includes Barman with version 2.18 orhigher, CloudNativePG enables you to specify tags as key-value pairsfor backup objects, namely base backups, WAL files and history files.

You can use two properties in the .spec.backup.barmanObjectStore definition:

The excerpt of a YAML manifest below provides an example of usage of thisfeature:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
[...]
spec:
  backup:
    barmanObjectStore:
      [...]
      tags:
        backupRetentionPolicy: "expire"
      historyTags:
        backupRetentionPolicy: "keep"

Backup and Recovery¶

Cloud provider support¶

S3¶

Other S3-compatible Object Storages providers¶

MinIO Gateway¶

Azure Blob Storage¶

Other Azure Blob Storage compatible providers¶

Google Cloud Storage¶

Running inside Google Kubernetes Engine¶

Using authentication¶

On-demand backups¶

Scheduled backups¶

WAL archiving¶

Recovery¶

Retention policies¶

Compression algorithms¶

Tagging of backup objects¶

Previous topic

Next topic

Table of Contents