Features in detail ================== In this section we present several Barman features and discuss their applicability and the configuration required to use them. This list is not exhaustive, as many scenarios can be created working on the Barman configuration. Nevertheless, it is useful to discuss common patterns. Backup features --------------- Incremental backup ^^^^^^^^^^^^^^^^^^ Barman implements **file-level incremental backup**. Incremental backup is a type of full periodic backup which only saves data changes from the latest full backup available in the catalog for a specific PostgreSQL server. It must not be confused with differential backup, which is implemented by WAL continuous archiving. .. NOTE:: Block level incremental backup will be available in future versions. .. IMPORTANT:: The `reuse_backup` option can't be used with the `postgres` backup method at this time. The main goals of incremental backups in Barman are: - Reduce the time taken for the full backup process - Reduce the disk space occupied by several periodic backups (**data deduplication**) This feature heavily relies on ``rsync`` and `hard links `_ , which must therefore be supported by both the underlying operating system and the file system where the backup data resides. The main concept is that a subsequent base backup will share those files that have not changed since the previous backup, leading to relevant savings in disk usage. This is particularly true of VLDB contexts and of those databases containing a high percentage of read-only historical tables. Barman implements incremental backup through a global/server option called ``reuse_backup`` , that transparently manages the ``barman backup`` command. It accepts three values: - ``off`` : standard full backup (default) - ``link`` : incremental backup, by reusing the last backup for a server and creating a hard link of the unchanged files (for backup space and time reduction) - ``copy`` : incremental backup, by reusing the last backup for a server and creating a copy of the unchanged files (just for backup time reduction) The most common scenario is to set ``reuse_backup`` to ``link`` , as follows: .. code:: ini reuse_backup = link Setting this at global level will automatically enable incremental backup for all your servers. As a final note, users can override the setting of the ``reuse_backup`` option through the ``--reuse-backup`` runtime option for the ``barman backup`` command. Similarly, the runtime option accepts three values: ``off`` , ``link`` and ``copy`` . For example, you can run a one-off incremental backup as follows: .. code:: bash barman backup --reuse-backup=link Limiting bandwidth usage ^^^^^^^^^^^^^^^^^^^^^^^^ It is possible to limit the usage of I/O bandwidth through the ``bandwidth_limit`` option (global/per server), by specifying the maximum number of kilobytes per second. By default it is set to 0, meaning no limit. .. IMPORTANT:: the `bandwidth_limit` option is supported with the `postgres` backup method, but the `tablespace_bandwidth_limit` option is available only if you use `rsync` . In case you have several tablespaces and you prefer to limit the I/O workload of your backup procedures on one or more tablespaces, you can use the ``tablespace_bandwidth_limit`` option (global/per server): .. code:: ini tablespace_bandwidth_limit = tbname:bwlimit[, tbname:bwlimit, ...] The option accepts a comma separated list of pairs made up of the tablespace name and the bandwidth limit (in kilobytes per second). When backing up a server, Barman will try and locate any existing tablespace in the above option. If found, the specified bandwidth limit will be enforced. If not, the default bandwidth limit for that server will be applied. Network Compression ^^^^^^^^^^^^^^^^^^^ It is possible to reduce the size of transferred data using compression. It can be enabled using the ``network_compression`` option (global/per server): .. IMPORTANT:: the `network_compression` option is not available with the `postgres` backup method. .. code:: ini network_compression = true|false Setting this option to ``true`` will enable data compression during network transfers (for both backup and recovery). By default it is set to ``false`` . Backup Compression ^^^^^^^^^^^^^^^^^^ Barman can use the compression features of pg_basebackup in order to compress the backup data during the backup process. This can be enabled using the ``backup_compression`` config option (global/per server): .. IMPORTANT:: the `backup_compression` and other options discussed in this section are not available with the `rsync` or `local-rsync` backup methods. Only with `postgres` backup method. Compression algorithms ^^^^^^^^^^^^^^^^^^^^^^ Setting this option will cause pg_basebackup to compress the backup using the specified compression algorithm. Currently, supported algorithm in Barman are: ``gzip`` , ``lz4`` ,\ ``zstd`` and ``none`` . ``none`` compression algorithm will create an uncompressed archive. .. code:: ini backup_compression = gzip|lz4|zstd|none Barman requires the CLI utility for the selected compression algorithm to be available on both the Barman server and the PostgreSQL server. The CLI utility is used to extract the backup label from the compressed backup and to decompress the backup on the PostgreSQL server during recovery. These can be installed through system packages named ``gzip`` , ``lz4`` and ``zstd`` on Debian, Ubuntu, RedHat, CentOS and SLES systems. .. Note:: On Ubuntu 18.04 (bionic) the `lz4` utility is available in the `liblz4-tool` pacakge. .. Note:: `zstd` version must be 1.4.4 or higher. The system packages for `zstd` on Debian 10 (buster), Ubuntu 18.04 (bionic) and SLES 12 install an earlier version - `backup_compression = zstd` will not work with these packages. .. Note:: `lz4` and `zstd` are only available with PostgreSQL version 15 or higher. .. IMPORTANT:: If you are using `backup_compression` you must also set `recovery_staging_path` so that `barman recover` is able to recover the compressed backups. See the :ref:`Recovering compressed backups ` section for more information. Compression workers ^^^^^^^^^^^^^^^^^^^ This optional parameter allows compression using multiple threads to increase compression speed (default being 0). .. code:: ini backup_compression_workers = 2 .. Note :: This option is only available with `zstd` compression. .. Note :: `zstd` version must be 1.5.0 or higher. Or 1.4.4 or higher compiled with multithreading option. Compression level ^^^^^^^^^^^^^^^^^ The compression level can be specified using the ``backup_compression_level`` option. This should be set to an integer value supported by the compression algorithm specified in ``backup_compression`` . If not defined, compression algorithm default value will be used. ``none`` compression only supports ``backup_compression_level=0`` . .. Note :: `backup_compression_level` available and default values depends on the compression algorithm used. Please check the compression algorithm documentation for more details. .. Note :: On PostgreSQL version prior to 15, `gzip` support `backup_compression_level=0` . It results using default compression level Compression location ^^^^^^^^^^^^^^^^^^^^ When using Barman with PostgreSQL version 15 or higher it is possible to specify for compression to happen on the server (i.e. PostgreSQL will compress the backup) or on the client (i.e. pg_basebackup will compress the backup). This can be achieved using the ``backup_compression_location`` option: .. IMPORTANT:: the `backup_compression_location` option is only available when running against PostgreSQL 15 or later. .. code:: ini backup_compression_location = server|client Using ``backup_compression_location = server`` should reduce the network bandwidth required by the backup at the cost of moving the compression work onto the PostgreSQL server. When ``backup_compression_location`` is set to ``server`` then an additional option, ``backup_compression_format`` , can be set to ``plain`` in order to have pg_basebackup uncompress the data before writing it to disk: Compression format ^^^^^^^^^^^^^^^^^^ .. code:: ini backup_compression_format = plain|tar If ``backup_compression_format`` is unset or has the value ``tar`` then the backup will be written to disk as compressed tarballs. A description of both the ``plain`` and ``tar`` formats can be found in the [pg_basebackup documentation][pg_basebackup-documentation]. .. IMPORTANT :: Barman uses external tools to manage compressed backups. Depending on the `backup_compression` and `backup_compression_format` You may need to install one or more tools on the Postgres server and the Barman server. The following table will help you choose according to your configuration. .. csv-table:: :header: **backup_compression**,**backup_compression_format**,**Postgres server**,**Barman server** :widths: 15,12,12,20 :align: left :class: longtable gzip,plain,**tar** ,None gzip,tar,**tar** ,**tar** lz4,plain,"**tar, lz4** ",None lz4,tar,"**tar, lz4** ","**tar, lz4** " zstd,plain,"**tar, zstd** ",None zstd,tar,"**tar, zstd** ","**tar, zstd** " none,tar,**tar** ,**tar** Concurrent backup ^^^^^^^^^^^^^^^^^ Normally, during backup operations, Barman uses PostgreSQL native functions ``pg_start_backup`` and ``pg_stop_backup`` for *concurrent backup*. ``[^ABOUT_CONCURRENT_BACKUP]`` This is the recommended way of taking backups for PostgreSQL 9.6 and above (though note the functions have been renamed to ``pg_backup_start`` and ``pg_backup_stop`` in the PostgreSQL 15 beta). As well as being the recommended backup approach, concurrent backup also allows the following architecture scenario with Barman: **backup from a standby server**, using ``rsync`` . By default, ``backup_options`` is set to ``concurrent_backup`` . If exclusive backup is required for PostgreSQL servers older than version 15 then users should set ``backup_options`` to ``exclusive_backup`` . When ``backup_options`` is set to ``concurrent_backup`` , Barman activates the concurrent backup mode for a server and follows these two simple rules: - ``ssh_command`` must point to the destination Postgres server - ``conninfo`` must point to a database on the destination Postgres database. .. IMPORTANT :: In case of a concurrent backup, currently Barman cannot determine whether the closing WAL file of a full backup has actually been shipped - opposite of an exclusive backup where PostgreSQL itself makes sure that the WAL file is correctly archived. Be aware that the full backup cannot be considered consistent until that WAL file has been received and archived by Barman. Barman 2.5 introduces a new state, called `WAITING_FOR_WALS` , which is managed by the `check-backup` command (part of the ordinary maintenance job performed by the `cron` command). From Barman 2.10, you can use the `--wait` option with `barman backup` command. Concurrent backup of a standby ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If backing up a standby then the following `configuration options `_ should point to the standby server: - ``conninfo`` - ``streaming_conninfo`` (when using ``backup_method = postgres`` or ``streaming_archiver = on`` ) - ``ssh_command`` (when using ``backup_method = rsync`` ) The following config option should point to the primary server: - ``primary_conninfo`` Barman will use ``primary_conninfo`` to switch to a new WAL on the primary so that the concurrent backup against the standby can complete without having to wait for a WAL switch to occur naturally. .. NOTE :: It is especially important that `primary_conninfo` is set if the standby is to be backed up when there is little or no write traffic on the primary. As of Barman 3.8.0, If ``primary_conninfo`` is set, is possible to add for a server a ``primary_checkpoint_timeout`` option, which is the maximum time (in seconds) for Barman to wait for a new WAL file to be produced before forcing the execution of a checkpoint on the primary. The ``primary_checkpoint_timeout`` option should be set to an amount of seconds greater of the value of the ``archive_timeout`` option set on the primary server. If ``primary_conninfo`` is not set then the backup will still run however it will wait at the stop backup stage until the current WAL segment on the primary is newer than the latest WAL required by the backup. Barman currently requires that WAL files and backup data come from the same PostgreSQL server. In the case that the standby is promoted to primary the backups and WALs will continue to be valid however you may wish to update the Barman configuration so that it uses the new standby for taking backups and receiving WALs. WALs can be obtained from the standby using either WAL streaming or WAL archiving. To use WAL streaming follow the instructions in the :ref:`WAL streaming ` section. To use WAL archiving from the standby follow the instructions in the :ref:`WAL archiving via archive_command <#wal-archiving-via-archive_command>` section *and additionally* set ``archive_mode = always`` in the PostgreSQL config on the standby server. .. NOTE :: With PostgreSQL 10 and earlier Barman cannot handle WAL streaming and WAL archiving being enabled at the same time on a standby. You must therefore disable WAL archiving if using WAL streaming and vice versa. This is because it is possible for WALs produced by PostgreSQL 10 and earlier to be logically equivalent but differ at the binary level, causing Barman to fail to detect that two WALs are identical. Immediate checkpoint ^^^^^^^^^^^^^^^^^^^^ Before starting a backup, Barman requests a checkpoint, which generates additional workload. Normally that checkpoint is throttled according to the settings for workload control on the PostgreSQL server, which means that the backup could be delayed. This default behaviour can be changed through the ``immediate_checkpoint`` configuration global/server option (set to ``false`` by default). If ``immediate_checkpoint`` is set to ``true`` , PostgreSQL will not try to limit the workload, and the checkpoint will happen at maximum speed, starting the backup as soon as possible. At any time, you can override the configuration option behaviour, by issuing ``barman backup`` with any of these two options: - ``--immediate-checkpoint`` , which forces an immediate checkpoint; - ``--no-immediate-checkpoint`` , which forces to wait for the checkpoint to happen. Local backup ^^^^^^^^^^^^ .. Note :: This feature is not recommended for production usage, as Barman and PostgreSQL reside on the same server and are part of the same single point of failure. Some EnterpriseDB customers have requested to add support for local backup to Barman to be used under specific circumstances and, most importantly, under the 24/7 production service delivered by the company. Using this feature currently requires installation from sources, or to customise the environment for the `postgres` user in terms of permissions as well as logging and cron configurations. Under special circumstances, Barman can be installed on the same server where the PostgreSQL instance resides, with backup data stored on a separate volume from PGDATA and, where applicable, tablespaces. Usually, these volumes reside on network storage appliances, with filesystems like NFS. This architecture is not endorsed by EnterpriseDB. For an enhanced business continuity experience of PostgreSQL, with better results in terms of RPO and RTO, EnterpriseDB still recommends the shared nothing architecture with a remote installation of Barman, capable of acting like a witness server for replication and monitoring purposes. The only requirement for local backup is that Barman runs with the same user as the PostgreSQL server, which is normally ``postgres`` . Given that the Community packages by default install Barman under the ``barman`` user, this use case requires manual installation procedures that include: - cron configurations - log configurations, including logrotate In order to use local backup for a given server in Barman, you need to set ``backup_method`` to ``local-rsync`` . The feature is essentially identical to its ``rsync`` equivalent, which relies on SSH instead and operates remotely. With ``local-rsync`` file system copy is performed issuing ``rsync`` commands locally (for this reason it is required that Barman runs with the same user as PostgreSQL). An excerpt of configuration for local backup for a server named ``local-pg13`` is: .. code:: ini [local-pg13] description = "Local PostgreSQL 13" backup_method = local-rsync ... Archiving features ------------------ WAL compression ^^^^^^^^^^^^^^^ The ``barman cron`` command will compress WAL files if the ``compression`` option is set in the configuration file. This option allows five values: - ``bzip2`` : for Bzip2 compression (requires the ``bzip2`` utility) - ``gzip`` : for Gzip compression (requires the ``gzip`` utility) - ``pybzip2`` : for Bzip2 compression (uses Python’s internal compression module) - ``pygzip`` : for Gzip compression (uses Python’s internal compression module) - ``pigz`` : for Pigz compression (requires the ``pigz`` utility) - ``custom`` : for custom compression, which requires you to set the following options as well: - ``custom_compression_filter`` : a compression filter - ``custom_decompression_filter`` : a decompression filter - ``custom_compression_magic`` : a hex string to identify a custom compressed wal file NOTE: All methods but ``pybzip2`` and ``pygzip`` require ``barman archive-wal`` to fork a new process. Synchronous WAL streaming ^^^^^^^^^^^^^^^^^^^^^^^^^ Barman can also reduce the Recovery Point Objective to zero, by collecting the transaction WAL files like a synchronous standby server would. To configure such a scenario, the Barman server must be configured to archive WALs via the :ref:`PostgreSQL streaming connection ` , and the ``receive-wal`` process should figure as a synchronous standby of the PostgreSQL server. First of all, you need to retrieve the application name of the Barman ``receive-wal`` process with the ``show-servers`` command: .. code:: bash barman@backup$ barman show-servers pg|grep streaming_archiver_name streaming_archiver_name: barman_receive_wal Then the application name should be added to the ``postgresql.conf`` file as a synchronous standby: .. code:: ini synchronous_standby_names = barman_receive_wal .. IMPORTANT :: this is only an example of configuration, to show you that Barman is eligible to be a synchronous standby node. We are not suggesting to use ONLY Barman. You can read `Synchronous Replication `_ from the PostgreSQL documentation for further information on this topic. The PostgreSQL server needs to be restarted for the configuration to be reloaded. If the server has been configured correctly, the ``replication-status`` command should show the ``receive_wal`` process as a synchronous streaming client: .. code:: bash [root@backup ~]# barman replication-status pg Status of streaming clients for server pg: Current xlog location on master: 0/9000098 Number of streaming clients: 1 1. #1 Sync WAL streamer Application name: barman_receive_wal Sync stage : 3/3 Remote write Communication : TCP/IP IP Address : 139.59.135.32 / Port: 58262 / Host: - User name : streaming_barman Current state : streaming (sync) Replication slot: barman WAL sender PID : 2501 Started at : 2016-09-16 10:33:01.725883+00:00 Sent location : 0/9000098 (diff: 0 B) Write location : 0/9000098 (diff: 0 B) Flush location : 0/9000098 (diff: 0 B) Catalog management features --------------------------- Minimum redundancy safety ^^^^^^^^^^^^^^^^^^^^^^^^^ You can define the minimum number of periodic backups for a PostgreSQL server, using the global/per server configuration option called ``minimum_redundancy`` , by default set to 0. By setting this value to any number greater than 0, Barman makes sure that at any time you will have at least that number of backups in a server catalog. This will protect you from accidental ``barman delete`` operations. .. IMPORTANT:: Make sure that your retention policy settings do not collide with minimum redundancy requirements. Regularly check Barman's log for messages on this topic. Retention policies ^^^^^^^^^^^^^^^^^^ Barman supports **retention policies** for backups. A backup retention policy is a user-defined policy that determines how long backups and related archive logs (Write Ahead Log segments) need to be retained for recovery procedures. Based on the user’s request, Barman retains the periodic backups required to satisfy the current retention policy and any archived WAL files required for the complete recovery of those backups. Barman users can define a retention policy in terms of **backup redundancy** (how many periodic backups) or a **recovery window** (how long). Retention policy based on redundancy In a redundancy based retention policy, the user determines how many periodic backups to keep. A redundancy-based retention policy is contrasted with retention policies that use a recovery window. Retention policy based on recovery window A recovery window is one type of Barman backup retention policy, in which the DBA specifies a period of time and Barman ensures retention of backups and/or archived WAL files required for point-in-time recovery to any time during the recovery window. The interval always ends with the current time and extends back in time for the number of days specified by the user. For example, if the retention policy is set for a recovery window of seven days, and the current time is 9:30 AM on Friday, Barman retains the backups required to allow point-in-time recovery back to 9:30 AM on the previous Friday. Scope ^^^^^ Retention policies can be defined for: - **PostgreSQL periodic base backups**: through the ``retention_policy`` configuration option - **Archive logs**, for Point-In-Time-Recovery: through the ``wal_retention_policy`` configuration option .. IMPORTANT:: In a temporal dimension, archive logs must be included in the time window of periodic backups. There are two typical use cases here: full or partial point-in-time recovery. Full point in time recovery scenario: Base backups and archive logs share the same retention policy, allowing you to recover at any point in time from the first available backup. Partial point in time recovery scenario: Base backup retention policy is wider than that of archive logs, for example allowing users to keep full, weekly backups of the last 6 months, but archive logs for the last 4 weeks (granting to recover at any point in time starting from the last 4 periodic weekly backups). .. IMPORTANT:: Currently, Barman implements only the **full point in time recovery** scenario, by constraining the `wal_retention_policy` option to `main` . How they work ^^^^^^^^^^^^^ Retention policies in Barman can be: - **automated**: enforced by ``barman cron`` - **manual**: Barman simply reports obsolete backups and allows you to delete them .. IMPORTANT:: Currently Barman does not implement manual enforcement. This feature will be available in future versions. Configuration and syntax ^^^^^^^^^^^^^^^^^^^^^^^^ Retention policies can be defined through the following configuration options: - ``retention_policy`` : for base backup retention - ``wal_retention_policy`` : for archive logs retention - ``retention_policy_mode`` : can only be set to ``auto`` (retention policies are automatically enforced by the ``barman cron`` command) These configuration options can be defined both at a global level and a server level, allowing users maximum flexibility on a multi-server environment. Syntax for ``retention_policy`` ''''''''''''''''''''''''''''''' The general syntax for a base backup retention policy through ``retention_policy`` is the following: .. code:: ini retention_policy = {REDUNDANCY value | RECOVERY WINDOW OF value {DAYS | WEEKS | MONTHS}} Where: - syntax is case insensitive - ``value`` is an integer and is > 0 - in case of **redundancy retention policy**: - ``value`` must be greater than or equal to the server minimum redundancy level (if that value is not assigned, a warning is generated) - the first valid backup is the value-th backup in a reverse ordered time series - in case of **recovery window policy**: - the point of recoverability is: current time - window - the first valid backup is the first available backup before the point of recoverability; its value in a reverse ordered time series must be greater than or equal to the server minimum redundancy level (if it is not assigned to that value and a warning is generated) By default, ``retention_policy`` is empty (no retention enforced). Syntax for ``wal_retention_policy`` ''''''''''''''''''''''''''''''''''' Currently, the only allowed value for ``wal_retention_policy`` is the special value ``main`` , that maps the retention policy of archive logs to that of base backups. Hook scripts ------------ Barman allows a database administrator to run hook scripts on these two events: - before and after a backup - before and after the deletion of a backup - before and after a WAL file is archived - before and after a WAL file is deleted There are two types of hook scripts that Barman can manage: - standard hook scripts - retry hook scripts The only difference between these two types of hook scripts is that Barman executes a standard hook script only once, without checking its return code, whereas a retry hook script may be executed more than once, depending on its return code. Specifically, when executing a retry hook script, Barman checks the return code and retries indefinitely until the script returns either ``SUCCESS`` (with standard return code ``0`` ), or ``ABORT_CONTINUE`` (return code ``62`` ), or ``ABORT_STOP`` (return code ``63`` ). Barman treats any other return code as a transient failure to be retried. Users are given more power: a hook script can control its workflow by specifying whether a failure is transient. Also, in case of a ‘pre’ hook script, by returning ``ABORT_STOP`` , users can request Barman to interrupt the main operation with a failure. Hook scripts are executed in the following order: 1. The standard ‘pre’ hook script (if present) 2. The retry ‘pre’ hook script (if present) 3. The actual event (i.e. backup operation, or WAL archiving), if retry ‘pre’ hook script was not aborted with ``ABORT_STOP`` 4. The retry ‘post’ hook script (if present) 5. The standard ‘post’ hook script (if present) The output generated by any hook script is written in the log file of Barman. .. NOTE:: Currently, `ABORT_STOP` is ignored by retry 'post' hook scripts. In these cases, apart from logging an additional warning, `ABORT_STOP` will behave like `ABORT_CONTINUE` . Backup scripts ^^^^^^^^^^^^^^ These scripts can be configured with the following global configuration options (which can be overridden on a per server basis): - ``pre_backup_script`` : hook script executed before a base backup, only once, with no check on the exit code - ``pre_backup_retry_script`` : retry hook script executed before a base backup, repeatedly until success or abort - ``post_backup_retry_script`` : retry hook script executed after a base backup, repeatedly until success or abort - ``post_backup_script`` : hook script executed after a base backup, only once, with no check on the exit code The script definition is passed to a shell and can return any exit code. Only in case of a retry script, Barman checks the return code (see the :ref:`hook script section <#hook_scripts>` ). The shell environment will contain the following variables: - ``BARMAN_BACKUP_DIR`` : backup destination directory - ``BARMAN_BACKUP_ID`` : ID of the backup - ``BARMAN_CONFIGURATION`` : configuration file used by Barman - ``BARMAN_ERROR`` : error message, if any (only for the ``post`` phase) - ``BARMAN_PHASE`` : phase of the script, either ``pre`` or ``post`` - ``BARMAN_PREVIOUS_ID`` : ID of the previous backup (if present) - ``BARMAN_RETRY`` : ``1`` if it is a retry script, ``0`` if not - ``BARMAN_SERVER`` : name of the server - ``BARMAN_STATUS`` : status of the backup - ``BARMAN_VERSION`` : version of Barman Backup delete scripts ^^^^^^^^^^^^^^^^^^^^^ Version **2.4** introduces pre and post backup delete scripts. As previous scripts, backup delete scripts can be configured within global configuration options, and it is possible to override them on a per server basis: - ``pre_delete_script`` : hook script launched before the deletion of a backup, only once, with no check on the exit code - ``pre_delete_retry_script`` : retry hook script executed before the deletion of a backup, repeatedly until success or abort - ``post_delete_retry_script`` : retry hook script executed after the deletion of a backup, repeatedly until success or abort - ``post_delete_script`` : hook script launched after the deletion of a backup, only once, with no check on the exit code The script is executed through a shell and can return any exit code. Only in case of a retry script, Barman checks the return code (see the upper section). Delete scripts uses the same environmental variables of a backup script, plus: - ``BARMAN_NEXT_ID`` : ID of the next backup (if present) WAL archive scripts ^^^^^^^^^^^^^^^^^^^ Similar to backup scripts, archive scripts can be configured with global configuration options (which can be overridden on a per server basis): - ``pre_archive_script`` : hook script executed before a WAL file is archived by maintenance (usually ``barman cron`` ), only once, with no check on the exit code - ``pre_archive_retry_script`` : retry hook script executed before a WAL file is archived by maintenance (usually ``barman cron`` ), repeatedly until it is successful or aborted - ``post_archive_retry_script`` : retry hook script executed after a WAL file is archived by maintenance, repeatedly until it is successful or aborted - ``post_archive_script`` : hook script executed after a WAL file is archived by maintenance, only once, with no check on the exit code The script is executed through a shell and can return any exit code. Only in case of a retry script, Barman checks the return code (see the upper section). Archive scripts share with backup scripts some environmental variables: - ``BARMAN_CONFIGURATION`` : configuration file used by Barman - ``BARMAN_ERROR`` : error message, if any (only for the ``post`` phase) - ``BARMAN_PHASE`` : phase of the script, either ``pre`` or ``post`` - ``BARMAN_SERVER`` : name of the server Following variables are specific to archive scripts: - ``BARMAN_SEGMENT`` : name of the WAL file - ``BARMAN_FILE`` : full path of the WAL file - ``BARMAN_SIZE`` : size of the WAL file - ``BARMAN_TIMESTAMP`` : WAL file timestamp - ``BARMAN_COMPRESSION`` : type of compression used for the WAL file WAL delete scripts ^^^^^^^^^^^^^^^^^^ Version **2.4** introduces pre and post WAL delete scripts. Similarly to the other hook scripts, wal delete scripts can be configured with global configuration options, and is possible to override them on a per server basis: - ``pre_wal_delete_script`` : hook script executed before the deletion of a WAL file - ``pre_wal_delete_retry_script`` : retry hook script executed before the deletion of a WAL file, repeatedly until it is successful or aborted - ``post_wal_delete_retry_script`` : retry hook script executed after the deletion of a WAL file, repeatedly until it is successful or aborted - ``post_wal_delete_script`` : hook script executed after the deletion of a WAL file The script is executed through a shell and can return any exit code. Only in case of a retry script, Barman checks the return code (see the upper section). WAL delete scripts use the same environmental variables as WAL archive scripts. Recovery scripts ^^^^^^^^^^^^^^^^ Version **2.4** introduces pre and post recovery scripts. As previous scripts, recovery scripts can be configured within global configuration options, and is possible to override them on a per server basis: - ``pre_recovery_script`` : hook script launched before the recovery of a backup, only once, with no check on the exit code - ``pre_recovery_retry_script`` : retry hook script executed before the recovery of a backup, repeatedly until success or abort - ``post_recovery_retry_script`` : retry hook script executed after the recovery of a backup, repeatedly until success or abort - ``post_recovery_script`` : hook script launched after the recovery of a backup, only once, with no check on the exit code The script is executed through a shell and can return any exit code. Only in case of a retry script, Barman checks the return code (see the upper section). Recovery scripts uses the same environmental variables of a backup script, plus: - ``BARMAN_DESTINATION_DIRECTORY`` : the directory where the new instance is recovered - ``BARMAN_TABLESPACES`` : tablespace relocation map (JSON, if present) - ``BARMAN_REMOTE_COMMAND`` : secure shell command used by the recovery (if present) - ``BARMAN_RECOVER_OPTIONS`` : recovery additional options (JSON, if present) Customization ------------- Lock file directory ^^^^^^^^^^^^^^^^^^^ Barman allows you to specify a directory for lock files through the ``barman_lock_directory`` global option. Lock files are used to coordinate concurrent work at global and server level (for example, cron operations, backup operations, access to the WAL archive, and so on.). By default (for backward compatibility reasons), ``barman_lock_directory`` is set to ``barman_home`` . .. TIP:: Users are encouraged to use a directory in a volatile partition, such as the one dedicated to run-time variable data (e.g. `/var/run/barman` ). Binary paths ^^^^^^^^^^^^ As of version 1.6.0, Barman allows users to specify one or more directories where Barman looks for executable files, using the global/server option ``path_prefix`` . If a ``path_prefix`` is provided, it must contain a list of one or more directories separated by colon. Barman will search inside these directories first, then in those specified by the ``PATH`` environment variable. By default the ``path_prefix`` option is empty. Integration with cluster management systems ------------------------------------------- Barman has been designed for integration with standby servers (with streaming replication or traditional file based log shipping) and high availability tools like (https://www.repmgr.org/) [repmgr]. From an architectural point of view, PostgreSQL must be configured to archive WAL files directly to the Barman server. Barman, thanks to the ``get-wal`` framework, can also be used as a WAL hub. For this purpose, you can use the ``barman-wal-restore`` script, part of the ``barman-cli`` package, with all your standby servers. The ``replication-status`` command allows you to get information about any streaming client attached to the managed server, in particular hot standby servers and WAL streamers. Configuration Models ^^^^^^^^^^^^^^^^^^^^ Configuration models define a set of overrides for configuration options. These overrides can be applied to Barman servers which are part of the same cluster as the config models. They can be useful when handling clustered environments, so you can change the configuration of a Barman server in response to failover events, for example. As an example, let’s say you have a PostgreSQL cluster with the following nodes: - ``pg-node-1`` : primary - ``pg-node-2`` : standby - ``pg-node-3`` : standby Assume you are backing up from the primary node, and have a configuration which includes the following options: .. code:: ini [my-barman-server] cluster = my-cluster conninfo = host=pg-node-1 user=barman database=postgres streaming_conninfo = host=pg-node-1 user=streaming_barman ; other options... You could, for example, have a configuration model for that cluster as follows: .. code:: ini [my-barman-server:backup-from-pg-node-2] cluster = my-cluster model = true conninfo = host=pg-node-2 user=barman database=postgres streaming_conninfo = host=pg-node-2 user=streaming_barman Which could be applied upon a failover from ``pg-node-1`` to ``pg-node-2`` with the following command, so you start backing up from the new primary node: .. code:: bash barman config-switch my-barman-server my-barman-server:backup-from-pg-node-2 That will override the cluster configuration options with the values defined in the selected model. - NOTE\*: not all options are configurable through models. Please refer to `section 5 of the 'man' page `_ to check settings which scope applies to models. - NOTE\*: you might be interested in checking `pg-backup-api `_ , which can start a REST API and listen for remote requests for executing ``barman`` commands, including ``barman config-switch`` . Parallel jobs ------------- By default, Barman uses only one worker for file copy during both backup and recover operations. Starting from version 2.2, it is possible to customize the number of workers that will perform file copy. In this case, the files to be copied will be equally distributed among all parallel workers. It can be configured in global and server scopes, adding these in the corresponding configuration file: .. code:: ini parallel_jobs = n where ``n`` is the desired number of parallel workers to be used in file copy operations. The default value is 1. In any case, users can override this value at run-time when executing ``backup`` or ``recover`` commands. For example, you can use 4 parallel workers as follows: .. code:: bash barman backup --jobs 4 server1 Or, alternatively: .. code:: bash barman backup --j 4 server1 Please note that this parallel jobs feature is only available for servers configured through ``rsync`` /SSH. For servers configured through streaming protocol, Barman will rely on ``pg_basebackup`` which is currently limited to only one worker. Parallel jobs and sshd MaxStartups ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Barman limits the rate at which parallel Rsync jobs are started in order to avoid exceeding the maximum number of concurrent unauthenticated connections allowed by the SSH server. This maximum is defined by the sshd parameter ``MaxStartups`` - if more than ``MaxStartups`` connections have been created but not yet authenticated then the SSH server may drop some or all of the connections resulting in a failed backup or recovery. The default value of sshd ``MaxStartups`` on most platforms is 10. Barman therefore starts parallel jobs in batches of 10 and does not start more than one batch of jobs within a one second time period. This yields an effective rate limit of 10 jobs per second. This limit can be changed using the following two configuration options: - ``parallel_jobs_start_batch_size`` : The maximum number of parallel jobs to start in a single batch. - ``parallel_jobs_start_batch_period`` : The time period in seconds over which a single batch of jobs will be started. For example, to ensure no more than five new Rsync jobs will be created within any two second time period: .. code:: ini parallel_jobs_start_batch_size = 5 parallel_jobs_start_batch_period = 2 The configuration options can be overridden using the following arguments with both ``barman backup`` and ``barman recover`` commands: - ``--jobs-start-batch-size`` - ``--jobs-start-batch-period`` Geographical redundancy ----------------------- It is possible to set up **cascading backup architectures** with Barman, where the source of a backup server is a Barman installation rather than a PostgreSQL server. This feature allows users to transparently keep geographically distributed copies of PostgreSQL backups. In Barman jargon, a backup server that is connected to a Barman installation rather than a PostgreSQL server is defined **passive node**. A passive node is configured through the ``primary_ssh_command`` option, available both at global (for a full replica of a primary Barman installation) and server level (for mixed scenarios, having both direct and passive servers). Sync information ^^^^^^^^^^^^^^^^ The ``barman sync-info`` command is used to collect information regarding the current status of a Barman server that is useful for synchronisation purposes. The available syntax is the following: .. code:: bash barman sync-info [--primary] [ []] The command returns a JSON object containing: - A map with all the backups having status ``DONE`` for that server - A list with all the archived WAL files - The configuration for the server - The last read position (in the xlog database file) - the name of the last read WAL file The JSON response contains all the required information for the synchronisation between the ``master`` and a ``passive`` node. If ``--primary`` is specified, the command is executed on the defined primary node, rather than locally. Configuration ^^^^^^^^^^^^^ Configuring a server as ``passive node`` is a quick operation. Simply add to the server configuration the following option: .. code:: ini primary_ssh_command = ssh barman@primary_barman This option specifies the SSH connection parameters to the primary server, identifying the source of the backup data for the passive server. If you are invoking barman with the ``-c/--config`` option and you want to use the same option when the passive node invokes barman on the primary node then add the following option: .. code:: ini forward_config_path = true Node synchronisation ^^^^^^^^^^^^^^^^^^^^ When a node is marked as ``passive`` it is treated in a special way by Barman: - it is excluded from standard maintenance operations - direct operations to PostgreSQL are forbidden, including ``barman backup`` Synchronisation between a passive server and its primary is automatically managed by ``barman cron`` which will transparently invoke: 1. ``barman sync-info --primary`` , in order to collect synchronisation information 2. ``barman sync-backup`` , in order to create a local copy of every backup that is available on the primary node 3. ``barman sync-wals`` , in order to copy locally all the WAL files available on the primary node Manual synchronisation ^^^^^^^^^^^^^^^^^^^^^^ Although ``barman cron`` automatically manages passive/primary node synchronisation, it is possible to manually trigger synchronisation of a backup through: .. code:: bash barman sync-backup Launching ``sync-backup`` barman will use the primary_ssh_command to connect to the master server, then if the backup is present on the remote machine, will begin to copy all the files using rsync. Only one single synchronisation process per backup is allowed. WAL files also can be synchronised, through: .. code:: bash barman sync-wals Cloud snapshot backups ---------------------- Snapshot backups are backups which consist of one or more snapshots of cloud storage volumes. A snapshot backup can be taken for a :ref:`suitable PostgreSQL server <#prerequisites-for-cloud-snapshots>` using either of the following commands: - ``barman backup`` with the :ref:`required configuration operations for snapshots <#configuration-for-snapshot-backups>` if a Barman server is being used to store WALs and backup metadata. - ``barman-cloud-backup`` with the required :ref:`command line arguments <#barman-cloud-backup-for-snapshots>` if there is no Barman server and instead a cloud object store is being used for WALs and backup metadata. Snapshot backup details ^^^^^^^^^^^^^^^^^^^^^^^ The high level process for taking a snapshot backup is as follows: 1. Barman carries out a series of pre-flight checks to validate the snapshot options, instance and disks. 2. Barman starts a backup using the `PostgreSQL backup API `_ . 3. The cloud provider API is used to trigger a snapshot for each specified disk. Barman will wait until the snapshot has reached the required state for guaranteeing application consistency before moving on to the next disk. 4. Additional provider-specific data, such as the device name for each disk, is saved to the backup metadata. 5. The mount point and mount options for each disk are saved in the backup metadata. 6. Barman stops the backup using the PostgreSQL backup API. The cloud provider API calls are made on the node where the backup command runs; this will be either the Barman server (when ``barman backup`` is used) or the PostgreSQL server (when ``barman-cloud-backup`` is used). The following pre-flight checks are carried out before each backup and also when ``barman check`` runs against a server configured for snapshot backups: - The compute instance specified by ``snapshot_instance`` and any provider-specific arguments exists. - The disks specified by ``snapshot_disks`` exist. - The disks specified by ``snapshot_disks`` are attached to ``snapshot_instance`` . - The disks specified by ``snapshot_disks`` are mounted on ``snapshot_instance`` . Recovering from a snapshot backup ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Barman will not currently perform a fully automated recovery from snapshot backups. This is because recovery from snapshots requires the provision and management of new infrastructure which is something better handled by dedicated infrastructure-as-code solutions such as Terraform. However, the ``barman recover`` command can still be used to validate the snapshot recovery instance, carry out post-recovery tasks such as checking the PostgreSQL configuration for unsafe options and set any required PITR options. It will also copy the backup_label file into place (since the backup label is not stored in any of the volume snapshots) and copy across any required WALs (unless the ``--get-wal`` recovery option is used, in which case it will configure the PostgreSQL ``restore_command`` to fetch the WALs). If restoring a backup made with ``barman-cloud-backup`` then the more limited :ref:`barman-cloud-restore for snapshots ` command should be used instead of ``barman recover`` . Recovery from a snapshot backup consists of the following steps: 1. Provision a new disk for each snapshot taken during the backup. 2. Provision a compute instance where each disk provisioned in step 1 is attached and mounted according to the backup metadata. 3. Use the :ref:`barman recover <#recover>` or :ref:`barman-cloud-restore for snapshots ` command to validate and finalize the recovery. Steps 1 and 2 are best handled by an existing infrastructure-as-code system however it is also possible to carry these steps out manually or using a custom script. The following resources may be helpful when carrying out these steps: - An example `recovery script for GCP `_ . - An example `runbook for Azure `_ . The above resources make assumptions about the backup/recovery environment and should not be considered suitable for production use without further customization. Once the recovery instance is provisioned and disks cloned from the backup snapshots are attached and mounted, run ``barman recover`` with the following additional arguments: - ``--remote-ssh-command`` : The ssh command required to log in to the recovery instance. - ``--snapshot-recovery-instance`` : The name of the recovery instance as required by the cloud provider. - Any additional arguments specific to the snapshot provider. For example: .. code:: bash barman recover SERVER_NAME BACKUP_ID REMOTE_RECOVERY_DIRECTORY \ --remote-ssh-command ssh USER@HOST \ --snapshot-recovery-instance INSTANCE_NAME Barman will automatically detect that the backup is a snapshot backup and check that the attached disks were cloned from the snapshots for that backup. Barman will then prepare PostgreSQL for recovery by copying the backup label and WALs into place and setting any required recovery options in the PostgreSQL configuration. The following additional ``barman recover`` arguments are available with the ``gcp`` provider: - ``--gcp-zone`` : The name of the availability zone in which the recovery instance is located. If not provided then Barman will use the value of ``gcp_zone`` set in the server config. The following additional ``barman recover`` arguments are available with the ``azure`` provider: - ``--azure-resource-group`` : The resource group to which the recovery instance belongs. If not provided then Barman will use the value of ``azure_resource_group`` set in the server config. The following additional ``barman recover`` arguments are available with the ``aws`` provider: - ``--aws-region`` : The AWS region in which the recovery instance is located. If not provided then Barman will use the value of ``aws_region`` set in the server config. Note the following ``barman recover`` arguments / config variables are unavailable when recovering snapshot backups: .. csv-table:: :header: **Command argument**,**Config variable** . :widths: 10,30 :align: left :class: longtable `--bwlimit`,`bandwidth_limit` `--jobs`,`parallel_jobs` `--recovery-staging-path`,`recovery_staging_path` `--tablespace`,N/A Backup metadata for snapshot backups ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Whether the recovery disks and instance are provisioned via infrastructure-as-code, ad-hoc automation or manually, it will be necessary to query Barman to find the snapshots required for a given backup. This can be achieved using :ref:`barman show-backup <#show-backup>` which will provide details for each snapshot in the backup. For example: .. code:: bash $ barman show-backup primary 20230123T131430 Backup 20230123T131430: Server Name : primary System Id : 7190784995399903779 Status : DONE PostgreSQL Version : 140006 PGDATA directory : /opt/postgres/data Snapshot information: provider : gcp project : project_id device_name : pgdata snapshot_name : barman-av-ubuntu20-primary-pgdata-20230123t131430 snapshot_project : project_id Mount point : /opt/postgres Mount options : rw,noatime device_name : tbs1 snapshot_name : barman-av-ubuntu20-primary-tbs1-20230123t131430 snapshot_project : project_id Mount point : /opt/postgres/tablespaces/tbs1 Mount options : rw,noatime The the ``--format=json`` option can be used when integrating with external tooling, e.g.: .. code:: bash $ barman --format=json show-backup primary 20230123T131430 ... "snapshots_info": { "provider": "gcp", "provider_info": { "project": "project_id" }, "snapshots": [ { "mount": { "mount_options": "rw,noatime", "mount_point": "/opt/postgres" }, "provider": { "device_name": "pgdata", "snapshot_name": "barman-av-ubuntu20-primary-pgdata-20230123t131430", "snapshot_project": "project_id" } }, { "mount": { "mount_options": "rw,noatime", "mount_point": "/opt/postgres/tablespaces/tbs1" }, "provider": { "device_name": "tbs1", "snapshot_name": "barman-av-ubuntu20-primary-tbs1-20230123t131430", "snapshot_project": "project_id", } } ] } ... For backups taken with ``barman-cloud-backup`` there is an analogous `barman-cloud-backup-show `_ command which can be used along with ``barman-cloud-backup-list`` to query the backup metadata in the cloud object store. The metadata available in ``snapshots_info/provider_info`` and ``snapshots_info/snapshots/*/provider`` varies by cloud provider as explained in the following sections. GCP provider-specific metadata ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following fields are available in ``snapshots_info/provider_info`` : - ``project`` : The GCP project ID of the project which owns the resources involved in backup and recovery. The following fields are available in ``snapshots_info/snapshots/*/provider`` : - ``device_name`` : The short device name with which the source disk for the snapshot was attached to the backup VM at the time of the backup. - ``snapshot_name`` : The name of the snapshot. - ``snapshot_project`` : The GCP project ID which owns the snapshot. Azure provider-specific metadata ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following fields are available in ``snapshots_info/provider_info`` : - ``subscription_id`` : The Azure subscription ID which owns the resources involved in backup and recovery. - ``resource_group`` : The Azure resource group to which the resources involved in the backup belong. The following fields are available in ``snapshots_info/snapshots/*/provider`` : - ``location`` : The Azure location of the disk from which the snapshot was taken. - ``lun`` : The LUN identifying the disk from which the snapshot was taken at the time of the backup. - ``snapshot_name`` : The name of the snapshot. AWS provider-specific metadata ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following fields are available in ``snapshots_info/provider_info`` : - ``account_id`` : The ID of the AWS account which owns the resources used to make the backup. - ``region`` : The AWS region in which the resources involved in backup are located. The following fields are available in ``snapshots_info/snapshots/*/provider`` : - ``device_name`` : The device to which the source disk was mapped on the backup VM at the time of the backup. - ``snapshot_id`` : The ID of the snapshot as assigned by AWS. - ``snapshot_name`` : The name of the snapshot.