Configuring HARP for Cluster Management

The HARP configuration file follows a standard YAML style formatting which has been simplified for readability. This file can be found in the /etc/harp directory by default, and is named config.yml

The configuration file location can be explicitly provided to all HARP executables with the -f/--config argument.

Standard Configuration

HARP essentially operates as three components:

  • HARP Manager

  • HARP Proxy

  • harpctl

Each of these use the same standard config.yml configuration format, which should always include the following sections:

  • cluster.name - The name of the cluster to target for all operations.

  • dcs - DCS driver and connection configuration for all endpoints.

Essentially this means a standard preamble will always be included for HARP operations, and will resemble this:

Other sections should be considered optional or specific to the named HARP component.

Cluster Name

The ``name`` entry under the cluster heading is required for all interaction with HARP. Each HARP cluster has a name for both disambiguation purposes and for labeling data within the DCS for the specific cluster.

HARP Managers will write information about the cluster here for consumption by HARP Proxy and harpctl. HARP Proxy services will direct traffic to nodes within this cluster. The harpctl management tool will interact with this cluster.

DCS Settings

Configuring the Consensus Layer is key to HARP functionality. Without the DCS, HARP has nowhere to store cluster metadata, cannot hold leadership elections, and so on. Therefore this portion of the configuration is required, though certain elements are optional.

All elements should be specified under a section named dcs with multiple supplementary entries that will be described here.

  • `driver` : Required type of consensus layer to use.

    Currently may be etcd or bdr. Support for bdr as a consensus layer is experimental. Using bdr as the consensus layer reduces the additional software for consensus storage, but expects a minimum of three full BDR member nodes to maintain quorum during database maintenance.

  • `endpoints` : Required list of connection strings to contact the DCS.

    Every node of the DCS should be listed here if possible. This ensures HARP will continue to function so long as a majority of the DCS is still operational and reachable via the network.

    Format when using etcd as the consensus layer is as follows:

Format when using the experimental bdr consensus layer is as follows:

  • `request_timeout` : Time in milliseconds to consider a request as failed.

    If HARP makes a request to the DCS and receives no response within this time period, it should consider the operation as failed. This may cause the issue to be logged as an error or retried, depending on the nature of the request. Default: 250.

The following DCS SSL settings only apply when driver: etcd is set in the configuration file.

  • `ssl` : Either on or off to enable SSL communication with the DCS.

    Default: off

  • `ssl_ca_file` : Client SSL Certificate Authority (CA) file.

  • `ssl_cert_file` : Client SSL certificate file.

  • `ssl_key_file` : Client SSL key file.

Example

Here is an example of how HARP should be configured to contact an etcd DCS consisting of three nodes:

HARP Manager Specific

Besides the generic service options required for all HARP components, Manager needs at least one more setting:

  • `log_level` : One of DEBUG, INFO, WARNING, ERROR, or CRITICAL

    which may alter the amount of log output from HARP services.

  • `name` : Required name of the Postgres node represented by this Manager.

    Since Manager can only represent a specific node, that node is named here and also serves to name this Manager. If this is a BDR node, it should match the value used at node creation when executing the bdr.create_node(node_name, ...) function and as reported by the bdr.local_node_summary.node_name view column. Alphanumeric characters and underscores only.

  • `start_command` : This can be used instead of the information in DCS for

    starting the database to be monitored. This is required if using bdr as the consensus layer.

  • `status_command` : This can be used instead of the information in DCS for

    the Harp Manager to determine whether or not the database is running. This is required if using bdr as the consensus layer.

  • `stop_command` : This can be used instead of the information in DCS for

    stopping the database.

Thus a complete configuration example for HARP Manager could resemble this:

Note that this is essentially the DCS contact information, any associated service customizations, the name of the cluster itself, and the name of the node. All other settings are associated with the node itself and is stored within the DCS.

Please read the section on Node Bootstrapping for more about specific node settings and initializing nodes to be managed by HARP Manager.

HARP Proxy Specific

Some configuration options are specific to HARP Proxy. These affect how the daemon itself operates, and thus are currently located in the config.yml file itself.

Proxy-based settings may be specified under a proxy heading, and include:

  • `location` : Required name of Location HARP Proxy should represent.

    HARP Proxy nodes are directly tied to the Location where they are running, as they always direct traffic to the current Lead Master node. This must be specified for any defined proxy.

  • `log_level` : One of DEBUG, INFO, WARNING, ERROR, or CRITICAL

which may alter the amount of log output from HARP services.

  • Default: INFO

  • `name` : Name of this specific Proxy.

Each Proxy node is named to ensure any associated statistics or operating state are available in status checks and other interactive events.

  • `type` : Specifies whether pgbouncer or the experimental built-in passthrough proxy will be used. All proxies must use the same proxy type. It is recommended to only experiment with the simple proxy in combination with the experimental BDR DCS.

May be pgbouncer or builtin.

  • Default: pgbouncer

  • `pgbouncer_bin_dir` : Directory where PgBouncer binaries are located.

As HARP utilizes PgBouncer binaries, it needs to know where these are located. This can be platform or distribution dependent, so has no default. The assumption is that the appropriate binaries are in the environment’s PATH variable otherwise.

Example

HARP Proxy requires the cluster name, DCS connection settings, location, and name of the proxy in operation. An example lies below:

All other attributes are obtained from the DCS upon Proxy startup.

Run-Time Directives

While it is possible to confige HARP Manager, HARP Proxy, or harpctl with a minimum of YAML in the config.yml file, some customizations are held within the DCS itself. These values must either initialized via bootstrap or set specifically with harpctl set directives.

This section will outline these, and how they may be specified.

Cluster Wide

Settings here should be set under a cluster YAML heading during bootstrap, or modified with a harpctl set cluster command.

  • `event_sync_interval` : Time in milliseconds to wait for synchronization.

When events occur within HARP, they do so asynchronously across the cluster. HARP Managers start operating immediately when they detect metadata changes, and HARP Proxies may pause traffic and start reconfiguring endpoints. This is a safety interval that is meant to roughly approximate the maximum amount of event time skew that may exist between all HARP components.

For example, suppose Node A goes offline and HARP Manager on Node B commonly receives this event 5 milliseconds before Node C. A setting of at least 5ms would then be necessary to ensure all HARP Manager services have received the event before they begin to process it.

This also applies to HARP Proxy.

Node Directives

Most node-oriented settings can be changed and subsequently applied while HARP Manager is active. These items are retained in the DCS after initial bootstrap, and thus may be modified without altering a configuration file.

Settings here should be set under a node YAML heading during bootstrap, or modified with a harpctl set node command.

  • `camo_enforcement` : Whether CAMO queue state should be strictly enforced.

When set to strict, HARP will never allow switchover or failover to a BDR CAMO partner node unless it is fully caught up with the entire CAMO queue at the time of the migration. When set to lag_only, only standard lag thresholds such as maximum_camo_lag are applied.

  • `dcs_reconnect_interval` : The interval, measured in ms, between attempts that a disconnected node tries to reconnect to the DCS.

  • Default 1000.

  • `dsn` : Required full connection string to the managed Postgres node.

This parameter applies equally to all HARP services and enables micro-architectures which run only one service per container.

  • `db_data_dir` : Required Postgres data directory.

This is required by HARP Manager to start, stop, or reload the Postgres service. It is also the default location for configuration files, which may be used at a later time for controlling promotion of streaming replicas.

  • `db_conf_dir` : Location of Postgres configuration files.

Some platforms prefer storing Postgres confgiuration files away from the Postgres data directory itself. In these cases, this should be set to that expected location.

  • `db_log_file` : Location of Postgres log file.

  • Default /tmp/pg_ctl.out

  • `fence_node_on_dcs_failure` : In the event HARP is unable to reach the DCS, several readiness keys and the leadership lease itself will expire. This will implicitly prevent a node from routing consideration. However, such a node is not officially fenced, and the manager will not stop monitoring the database if stop_database_when_fenced is set to false.

  • Default: False

  • `leader_lease_duration` : Amount of time in seconds the Lead Master

lease will persist if not refreshed. This allows any HARP Manager a certain grace period to refresh the lock, before expiration allows another node to obtain the Lead Master lock instead.

  • Default: 30

  • `lease_refresh_interval` : Amount of time in milliseconds between

refreshes of the Lead Master lease. This essentially controls the time between each series of checks HARP Manager performs against its assigned Postgres node, and when the status of the node is updated in the Consensus layer.

  • Default: 5000

  • `max_dcs_failures` : The amount of DCS request failures before marking a node as fenced according to fence_node_on_dcs_failure. This prevents transient communication disruptions from shutting down database nodes.

  • Default: 10

  • `maximum_lag` : Highest allowable variance (in bytes) between last

recorded LSN of previous Lead Master and this node before being allowed to take the Lead Master lock. This prevents nodes experiencing terminal amounts of lag from taking the Lead Master lock. Set to -1 to disable this check.

  • Default: 1048576 (1MB)

  • `maximum_camo_lag` : Highest allowable variance (in bytes) between last

received LSN and applied LSN between this node and its CAMO partner(s). This should only apply to clusters where CAMO is both available and enabled. Thus this only applies to BDR EE clusters where pg2q.enable_camo is set. Clusters with particularly stringent CAMO apply queue restrictions should set this very low, or even to 0 to avoid any unapplied CAMO transactions. Set to -1 to disable this check.

  • Default: 1048576 (1MB)

  • `ready_status_duration` : Amount of time in seconds the node’s readiness

status will persist if not refreshed. This is a failsafe that will remove a node from being contacted by HARP Proxy if the HARP Manager in charge if it stops operating.

  • Default: 30

  • `db_bin_dir` : Directory where Postgres binaries are located.

As HARP utilizes Postgres binaries, such as pg_ctl, it needs to know where these are located. This can be platform or distribution dependent, so has no default. The assumption is that the appropriate binaries are in the environment’s PATH variable otherwise.

  • `priority` : Any numeric value.

Any node where this option is set to -1 will be unable to take the Lead Master role, even when attempting to explicitly set the Lead Master using harpctl.

  • Default: 100

  • `stop_database_when_fenced` : Rather than simply removing a node from all possible routing, stop the database on a node when it is fenced. This is an extra safeguard to prevent data from other sources than HARP Proxy from reaching the database, or in case proxies are unable to disconnect clients for some other reason.

  • Default: False

  • `consensus_timeout` : Amount of milliseconds before aborting a read or

write to the consensus layer. In the event the consensus layer loses quorum or becomes unreachable, we want near-instant errors rather than infinite timeouts. This prevents blocking behavior in such cases. Note: When using bdr as the consensus layer, the highest recognized timeout is 1000ms.

  • Default: 250

  • `use_unix_socket` : Specifies that HARP Manager should prefer to use

unix sockets to connect to the database.

  • Default: False

All of these run-time directives can be modified via harpctl. Consider if we wished to decrease the lease_refresh_interval to 100ms on node1:

Proxy Directives

Certain settings to the Proxy can be changed while the service is active. These items are retained in the DCS after initial bootstrap, and thus may be modified without altering a configuration file. Many of these settings are direct mappings to their PgBouncer equivalent, and we will note these where relevant.

Settings here should be set under a proxies YAML heading during bootstrap, or modified with a harpctl set proxy command. Properties set via harpctl set proxy require a restart of the proxy.

  • `auth_file` : The full path to a PgBouncer-style userlist.txt file.

HARP Proxy will use this file to store a pgbouncer user which will have access to PgBouncer’s Admin database. This file may be used for other users as well. Proxy will modify this file to add and modify the password for the pgbouncer user.

  • Default /etc/harp/userlist.txt

  • `auth_type` : What type of Postgres authentication to use for password

matching. This is actually a PgBouncer setting and is not fully compatible with the Postgres pg_hba.conf capabilities. We recommend using md5, pam cert, or scram-sha-256.

  • Default md5

  • `auth_query` : Query to verify a user’s password with Postgres.

Direct access to pg_shadow requires admin rights. It’s preferable to use a non-superuser that calls a SECURITY DEFINER function instead. If using TPAexec to create a cluster, a function named pgbouncer_get_auth will be installed on all databases within the pg_catalog namespace to fulfill this purpose.

  • `auth_user` : If auth_user is set, then any user not specified in

auth_file will be queried through the auth_query query from pg_shadow in the database, using auth_user. The password of auth_user will be taken from auth_file.

  • `client_tls_ca_file` : Root certificate file to validate client

certificates. Requires client_tls_sslmode to be set.

  • `client_tls_cert_file` : Certificate for private key. Clients can

validate it. Requires client_tls_sslmode to be set.

  • `client_tls_key_file` : Private key for PgBouncer to accept client

connections. Requires client_tls_sslmode to be set.

  • `client_tls_protocols` : Which TLS protocol versions are allowed for

client connections. Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3. Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3), secure (tlsv1.2,tlsv1.3), legacy (all).

  • Default secure

  • `client_tls_sslmode` : Whether to enable client SSL functionality.

May be one of disable allow prefer require verify-ca verify-full.

  • Default disable

  • `database_name` : Required name that represents which database clients

will use when connecting to HARP Proxy. This is a stable endpoint that will not change and points to the current node, database name, port, etc., necessary to connect to the Lead Master. The global value * may be used here so all connections get directed to this target regardless of database name.

  • `default_pool_size` : The maximum amount of active connections to allow

per database / user combination. This is for connection pooling purposes, but will do nothing in session pooling mode. This is a PgBouncer setting.

  • Default 25

  • `ignore_startup_parameters` : By default, PgBouncer allows only

parameters it can keep track of in startup packets: client_encoding, datestyle, timezone, and standard_conforming_strings. All other parameters will raise an error. To allow other parameters, they can be specified here so that PgBouncer knows that they are handled by the admin and it can ignore them. It is often necessary to set this to extra_float_digits for Java applications to function properly.

  • Default extra_float_digits

  • `listen_address` : IP address(es) where Proxy should listen for

connections. Used by pgbouncer and builtin proxy.

  • Default 0.0.0.0

  • `listen_port` : System Port where Proxy should listen for connections.

Used by pgbouncer and builtin proxy.

  • Default 6432

  • `max_client_conn` : The total maximum amount of active client

connections that are allowed on the Proxy. This can be many orders of magnitude greater than default_pool_size, as these are all connections that have yet to be assigned a session, or have released a session for use by another client connection. This is a PgBouncer setting.

  • Default 100

  • `monitor_interval` : Time in seconds between Proxy checks of PgBouncer.

Since HARP Proxy manages PgBouncer as the actual connection management layer, it needs to periodically check various status and stats to verify it’s still operational. Some of this information may also be logged or registered to the DCS.

  • Default 5

  • `server_tls_protocols` : Which TLS protocol versions are allowed for

server connections. Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3. Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3), secure (tlsv1.2,tlsv1.3), legacy (all).

  • Default secure

  • `server_tls_sslmode` : Whether to enable server SSL functionality.

May be one of disable allow prefer require verify-ca verify-full.

  • Default disable

  • `session_transfer_mode` : Method by which to transfer sessions.

May be one of fast wait reconnect.

  • Default wait

  • `server_transfer_timeout` : The number of seconds Harp proxy will wait before giving up on a PAUSE and issuing a KILL command.

  • Default 30

The following two options only apply when using the built-in proxy.

  • `keepalive` : The number of seconds the built-in proxy will wait before sending a keepalive message to an idle leader connection.

  • Default 5

  • `timeout` : The number of seconds the built-in proxy will wait before giving up on connecting to the leader.

  • Default 1

When using harpctl to change any of these settings for all proxies, use the global keyword in place of the proxy name. Example: