Configuring HARP for Cluster Management

The HARP configuration file follows a standard YAML style formatting which hasbeen simplified for readability. This file can be found in the /etc/harpdirectory by default, and is named config.yml

The configuration file location can be explicitly provided to all HARP executables with the -f/--config argument.

Standard Configuration

HARP essentially operates as three components:

  • HARP Manager

  • HARP Proxy

  • harpctl

Each of these use the same standard config.yml configuration format, whichshould always include the following sections:

  • cluster.name - The name of the cluster to target for all operations.

  • dcs - DCS driver and connection configuration for all endpoints.

Essentially this means a standard preamble will always be included for HARPoperations, and will resemble this:

cluster:
   * name: mycluster

dcs:
   * ...

Other sections should be considered optional or specific to the named HARPcomponent.

Cluster Name

The ``name`` entry under the cluster heading is required for all interaction with HARP. Each HARP cluster has a name for both disambiguationpurposes and for labeling data within the DCS for the specific cluster.

HARP Managers will write information about the cluster here for consumption by HARP Proxy and harpctl. HARP Proxy services will direct traffic to nodes within this cluster. The harpctl management tool will interact with this cluster.

DCS Settings

Configuring the Consensus Layer is key to HARP functionality. Without the DCS, HARP has nowhere to store cluster metadata, cannot hold leadership elections, and so on. Therefore this portion of the configuration is required, though certain elements are optional.

All elements should be specified under a section named dcs with multiple supplementary entries that will be described here.

  • **`driver`**: Required type of consensus layer to use.

    • Currently may be etcd or bdr. Support for bdr as a consensus layer is

    • experimental. Using bdr as the consensus layer reduces the

    • additional software for consensus storage, but expects a minimum of three

    • full BDR member nodes to maintain quorum during database maintenance.

  • **`endpoints`**: Required list of connection strings to contact the DCS.

    • Every node of the DCS should be listed here if possible. This ensures HARP

    • will continue to function so long as a majority of the DCS is still

    • operational and reachable via the network.

  • **`request_timeout`**: Time in milliseconds to consider a request as failed.

    • If HARP makes a request to the DCS and receives no response within this time

    • period, it should consider the operation as failed. This may cause the issue

    • to be logged as an error or retried, depending on the nature of the request.

    • Default: 250.

  • **`ssl`**: Either `on` or `off` to enable SSL communication with the DCS.

    • Default: off

  • **`ssl_ca_file`**: Client SSL Certificate Authority (CA) file.

  • **`ssl_cert_file`**: Client SSL certificate file.

  • **`ssl_key_file`**: Client SSL key file.

Example

Here is an example of how HARP should be configured to contact an etcd DCSconsisting of three nodes:

dcs:
   * driver: etcd
   * endpoints:
   *   - host1:2379
   *   - host2:2379
   *   - host3:2379

HARP Manager Specific

Besides the generic service options required for all HARP components, Managerneeds at least one more setting:

  • **`log_level`**: One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`

    • which may alter the amount of log output from HARP services.

  • **`name`**: Required name of the Postgres node represented by this Manager.

    • Since Manager can only represent a specific node, that node is named here and

    • also serves to name this Manager. If this is a BDR node, it should match the

    • value used at node creation when executing the

    • bdr.create_node(node_name, ...) function and as reported by the

    • bdr.local_node_summary.node_name view column. Alphanumeric characters

    • and underscores only.

Thus a complete configuration example for HARP Manager could resemble this:

cluster:
   * name: mycluster

dcs:
   * driver: etcd
   * endpoints:
   *   - host1:2379
   *   - host2:2379
   *   - host3:2379

manager:
   * name: node1
   * log_level: INFO

Note that this is essentially the DCS contact information, any associated service customizations, the name of the cluster itself, and the name of thenode. All other settings are associated with the node itself and is storedwithin the DCS.

Please read the section on Node Bootstrapping for more aboutspecific node settings and initializing nodes to be managed by HARP Manager.

HARP Proxy Specific

Some configuration options are specific to HARP Proxy. These affect how thedaemon itself operates, and thus are currently located in the config.yml fileitself.

Proxy-based settings may be specified under a proxy heading, and include:

  • **`location`**: Required name of Location HARP Proxy should represent.

    • HARP Proxy nodes are directly tied to the Location where they are running, as

    • they always direct traffic to the current Lead Master node. This must be

    • specified for any defined proxy.

  • **`log_level`**: One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, or `CRITICAL`

    • which may alter the amount of log output from HARP services.

      • Default: INFO

  • **`name`**: Name of this specific Proxy.

    • Each Proxy node is named to ensure any associated statistics or operating

    • state are available in status checks and other interactive events.

  • **`type`**: Specifies whether pgbouncer or the experimental simple passthrough proxy will be used.  All proxies must use the same proxy type.  It is recommended to only experiment with the simple proxy in combination with the experimental BDR DCS.

    • May be pgbouncer or simpleproxy.

      • Default: pgbouncer

  • **`pgbouncer_bin_dir`**: Directory where PgBouncer binaries are located.

    • As HARP utilizes PgBouncer binaries, it needs to know where these are

    • located. This can be platform or distribution dependent, so has no

    • default. The assumption is that the appropriate binaries are in the

    • environment’s PATH variable otherwise.

Example

HARP Proxy requires the cluster name, DCS connection settings, location, andname of the proxy in operation. An example lies below:

cluster:
   * name: mycluster

dcs:
   * driver: etcd
   * endpoints:
   *   - host1:2379
   *   - host2:2379
   *   - host3:2379

proxy:
   * name: proxy1
   * location: dc1
   * pgbouncer_bin_dir: /usr/sbin

All other attributes are obtained from the DCS upon Proxy startup.

Run-Time Directives

While it is possible to confige HARP Manager, HARP Proxy, or harpctl with aminimum of YAML in the config.yml file, some customizations are held withinthe DCS itself. These values must either initialized via bootstrap or setspecifically with harpctl set directives.

This section will outline these, and how they may be specified.

Cluster Wide

Settings here should be set under a cluster YAML heading during bootstrap, ormodified with a harpctl set cluster command.

  • **`event_sync_interval`**: Time in milliseconds to wait for synchronization.

    • When events occur within HARP, they do so asynchronously across the cluster.

    • HARP Managers start operating immediately when they detect metadata changes,

    • and HARP Proxies may pause traffic and start reconfiguring endpoints. This is

    • a safety interval that is meant to roughly approximate the maximum amount of

    • event time skew that may exist between all HARP components.

    • For example, suppose Node A goes offline and HARP Manager on Node B commonly

    • receives this event 5 milliseconds before Node C. A setting of at least 5ms

    • would then be necessary to ensure all HARP Manager services have received the

    • event before they begin to process it.

    • This also applies to HARP Proxy.

Node Directives

Most node-oriented settings can be changed and subsequently applied while HARP Manager is active. These items are retained in the DCS after initial bootstrap, and thus may be modified without altering a configuration file.

Settings here should be set under a node YAML heading during bootstrap, ormodified with a harpctl set node command.

  • **`camo_enforcement`**: Whether CAMO queue state should be strictly enforced.

    • When set to strict, HARP will never allow switchover or failover to a BDR

    • CAMO partner node unless it is fully caught up with the entire CAMO queue at

    • the time of the migration. When set to lag_only, only standard lag

    • thresholds such as maximum_camo_lag are applied.

  • **`dcs_reconnect_interval`**: The interval, measured in ms, between attempts that a disconnected node tries to reconnect to the DCS.

      • Default 1000.

  • **`dsn`**: Required full connection string to the managed Postgres node.

    • This parameter applies equally to all HARP services and enables

    • micro-architectures which run only one service per container.

    • !!! Note

    • HARP sets the `sslmode` argument to `require` by default and will prevent       connections to servers that do not require SSL. To disable this behavior,       explicitly set this parameter to a more permissive value such as       `disable`, `allow`, or `prefer`.

  • **`db_data_dir`**: Required Postgres data directory.

    • This is required by HARP Manager to start, stop, or reload the Postgres

    • service. It is also the default location for configuration files, which may

    • be used at a later time for controlling promotion of streaming replicas.

  • **`db_conf_dir`**: Location of Postgres configuration files.

    • Some platforms prefer storing Postgres confgiuration files away from the

    • Postgres data directory itself. In these cases, this should be set to that

    • expected location.

      • Default db_data_dir - ``db_log_file``: Location of Postgres log file. * Default /tmp/pg_ctl.out

  • **`fence_node_on_dcs_failure`**: In the event HARP is unable to reach the DCS, several readiness keys and the leadership lease itself will expire. This will implicitly prevent a node from routing consideration. However, such a node is not officially fenced, and the manager will not stop monitoring the database if `stop_database_when_fenced` is set to false.

      • Default: False - ``leader_lease_duration``: Amount of time in seconds the Lead Master lease will persist if not refreshed. This allows any HARP Manager a certain grace period to refresh the lock, before expiration allows another node to obtain the Lead Master lock instead.

      • Default: 30

  • **`lease_refresh_interval`**: Amount of time in milliseconds between

    • refreshes of the Lead Master lease. This essentially controls the time

    • between each series of checks HARP Manager performs against its assigned

    • Postgres node, and when the status of the node is updated in the Consensus

    • layer.

      • Default: 5000- ``max_dcs_failures``: The amount of DCS request failures before marking a node as fenced according to fence_node_on_dcs_failure. This prevents transient communication disruptions from shutting down database nodes.

      • Default: 10 - ``maximum_lag``: Highest allowable variance (in bytes) between last recorded LSN of previous Lead Master and this node before being allowed to take the Lead Master lock. This prevents nodes experiencing terminal amounts of lag from taking the Lead Master lock. Set to -1 to disable this check.

      • Default: 1048576 (1MB)

  • **`maximum_camo_lag`**: Highest allowable variance (in bytes) between last

    • received LSN and applied LSN between this node and its CAMO partner(s).

    • This should only apply to clusters where CAMO is both available and enabled.

    • Thus this only applies to BDR EE clusters where pg2q.enable_camo is set.

    • Clusters with particularly stringent CAMO apply queue restrictions should set

    • this very low, or even to 0 to avoid any unapplied CAMO transactions. Set to

    • -1 to disable this check.

      • Default: 1048576 (1MB)

  • **`ready_status_duration`**: Amount of time in seconds the node's readiness

    • status will persist if not refreshed. This is a failsafe that will remove a

    • node from being contacted by HARP Proxy if the HARP Manager in charge if it

    • stops operating.

      • Default: 30

  • **`db_bin_dir`**: Directory where Postgres binaries are located.

    • As HARP utilizes Postgres binaries, such as pg_ctl, it needs to know where

    • these are located. This can be platform or distribution dependent, so has no

    • default. The assumption is that the appropriate binaries are in the

    • environment’s PATH variable otherwise.

  • **`priority`**: Any numeric value greater than or equal to 0.

    • In the case two nodes have an equal amount of lag and other qualified

    • criteria to take the Lead Master lease, this acts as an additional ranking

    • value to prioritize one node over another. Any node where this option is set

    • to 0 will be unable to take the Lead Master role, even when attempting to

    • explicitly set the Lead Master using harpctl.

      • Default: 100

  • **`safety_interval`**: Time in milliseconds required before allowing routing

    • to a newly promoted Lead Master. This is intended to allow automated checks

    • against HARP Router to fail across the cluster before transitioning new

    • connections to the promoted node. This helps enforce fully synchronized

    • routing targets. 0 to disable.

      • Default: 100

  • **`stop_database_when_fenced`**: Rather than simply removing a node from all possible routing, stop the database on a node when it is fenced. This is an extra safeguard to prevent data from other sources than HARP Proxy from reaching the database, or in case proxies are unable to disconnect clients for some other reason.

      • Default: False - ``consensus_timeout``: Amount of milliseconds before aborting a read or write to the consensus layer. In the event the consensus layer loses quorum or becomes unreachable, we want near-instant errors rather than infinite timeouts. This prevents blocking behavior in such cases. Note: When using bdr as the consensus layer, the highest recognized timeout is 1000ms.

      • Default: 250

All of these run-time directives can be modified via harpctl. Consider if wewished to decrease the lease_refresh_interval to 100ms on node1:

harpctl set node node1 lease_refresh_interval=100

Proxy Directives

Certain settings to the Proxy can be changed while the service is active. These items are retained in the DCS after initial bootstrap, and thus may be modified without altering a configuration file. Many of these settings are direct mappings to their PgBouncer equivalent, and we will note these where relevant.

Settings here should be set under a proxies YAML heading during bootstrap, ormodified with a harpctl set proxy command.

  • **`auth_file`**: The full path to a PgBouncer-style `userlist.txt` file.

    • HARP Proxy will use this file to store a pgbouncer user which will have

    • access to PgBouncer’s Admin database. This file may be used for other users

    • as well. Proxy will modify this file to add and modify the password for the

    • pgbouncer user.

      • Default /etc/harp/userlist.txt

  • **`auth_type`**: What type of Postgres authentication to use for password

    • matching. This is actually a PgBouncer setting and is not fully compatible

    • with the Postgres pg_hba.conf capabilities. We recommend using md5, pam

    • cert, or scram-sha-256.

      • Default md5

  • **`auth_query`**: Query to verify a user’s password with Postgres.

    • Direct access to pg_shadow requires admin rights. It’s preferable to use a

    • non-superuser that calls a SECURITY DEFINER function instead. If using

    • TPAexec to create a cluster, a function named pgbouncer_get_auth will be

    • installed on all databases within the pg_catalog namespace to fulfill this

    • purpose.

  • **`auth_user`**: If `auth_user` is set, then any user not specified in

    • auth_file will be queried through the auth_query query from pg_shadow

    • in the database, using auth_user. The password of auth_user will be

    • taken from auth_file.

  • **`client_tls_ca_file`**: Root certificate file to validate client

    • certificates. Requires client_tls_sslmode to be set.

  • **`client_tls_cert_file`**: Certificate for private key. Clients can

    • validate it. Requires client_tls_sslmode to be set.

  • **`client_tls_key_file`**: Private key for PgBouncer to accept client

    • connections. Requires client_tls_sslmode to be set.

  • **`client_tls_protocols`**: Which TLS protocol versions are allowed for

    • client connections.

    • Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3.

    • Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3),

    • secure (tlsv1.2,tlsv1.3), legacy (all).

      • Default secure

  • **`client_tls_sslmode`**: Whether to enable client SSL functionality.

    • May be one of disable allow prefer require verify-ca verify-full.

      • Default disable

  • **`database_name`**: Required name that represents which database clients

    • will use when connecting to HARP Proxy. This is a stable endpoint that will

    • not change and points to the current node, database name, port, etc.,

    • necessary to connect to the Lead Master. The global value * may be used

    • here so all connections get directed to this target regardless of database

    • name.

  • **`default_pool_size`**: The maximum amount of active connections to allow

    • per database / user combination. This is for connection pooling purposes,

    • but will do nothing in session pooling mode. This is a PgBouncer setting.

      • Default 25

  • **`ignore_startup_parameters`**: By default, PgBouncer allows only

    • parameters it can keep track of in startup packets: client_encoding,

    • datestyle, timezone, and standard_conforming_strings. All other

    • parameters will raise an error. To allow other parameters, they can be

    • specified here so that PgBouncer knows that they are handled by the admin

    • and it can ignore them. It is often necessary to set this to

    • extra_float_digits for Java applications to function properly.

      • Default extra_float_digits

  • **`listen_address`**: IP address(es) where Proxy should listen for

    • connections. This is a PgBouncer setting.

      • Default 0.0.0.0

  • **`listen_port`**: System Port where Proxy should listen for connections.

    • This is a PgBouncer setting.

      • Default 6432

  • **`max_client_conn`**: The total maximum amount of active client

    • connections that are allowed on the Proxy. This can be many orders of

    • magnitude greater than default_pool_size, as these are all connections that

    • have yet to be assigned a session, or have released a session for use by

    • another client connection. This is a PgBouncer setting.

      • Default 100

  • **`monitor_interval`**: Time in seconds between Proxy checks of PgBouncer.

    • Since HARP Proxy manages PgBouncer as the actual connection management

    • layer, it needs to periodically check various status and stats to verify

    • it’s still operational. Some of this information may also be logged or

    • registered to the DCS.

      • Default 5

  • **`server_tls_protocols`**: Which TLS protocol versions are allowed for

    • server connections.

    • Allowed values: tlsv1.0, tlsv1.1, tlsv1.2, tlsv1.3.

    • Shortcuts: all (tlsv1.0,tlsv1.1,tlsv1.2,tlsv1.3),

    • secure (tlsv1.2,tlsv1.3), legacy (all).

      • Default secure

  • **`server_tls_sslmode`**: Whether to enable server SSL functionality.

    • May be one of disable allow prefer require verify-ca verify-full.

      • Default disable

  • **`session_transfer_mode`**: Method by which to transfer sessions.

    • May be one of fast wait reconnect.

      • Default wait

  • **`server_transfer_timeout`**: The number of seconds Harp proxy will wait before giving up on a PAUSE and issuing a KILL command.

      • Default 30

When using harpctl to change any of these settings for all proxies, use the global keyword in place of the proxy name. Example:

harpctl set proxy global max_client_conn 1000