Group Commit

The goal of Group Commit is to protect against data loss in case of single node failures or temporary outages. You achieve this by requiring more than one BDR node to successfully receive and confirm a transaction at COMMIT time.

Requirements

During normal operation, Group Commit is completely transparent to the application. Upon failover, the reconciliation phase needs to be explicitly triggered or consolidated by either the application or a proxy in between. HARP provides native support for Group Commit and triggers the reconciliation phase, making this equally transparent to the client.

On the origin node, a transaction committed with Group Commit uses two-phase commit underneath. Therefore, configure max_prepared_transactions high enough to handle all such transactions originating per node.

Configuration

To use Group Commit, first define a commit scope. This determines the BDR nodes involved in the commit of a transaction. Once a scope is established, you can configure a transaction to use Group Commit as follows:

BEGIN;
SET LOCAL bdr.commit_scope = example_scope;
...
COMMIT;

For this example, you might previously have defined the commit scope as:

SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := example_bdr_group,
    rule := ANY 2 (example_bdr_group)
);

This assumes a node group named example_bdr_group exists and includes at least two BDR nodes as members, either directly or in subgroups. Any transaction committed in the example_scope requires one extra confirmation from a BDR node in the group. Together with the origin node, this accounts for “ANY 2” nodes out of the group, on which the transaction is guaranteed to be visible and durable after the commit.

Origin groups

Rules for commit scopes can depend on the node the transaction is committed on, that is, the node that acts as the origin for the transaction. To make this transparent for the application, BDR allows a commit scope to define different rules depending on where the transaction originates from.

For example, consider a EDB Postgres Distributed cluster with nodes spread across two data centers: a left and a right one. Assume the top-level BDR node group is called top_group . You can use the following commands to set up subgroups and create a commit scope requiring all nodes in the local data center to confirm the transaction but only one node from the remote one.

- - create sub-groups
SELECT bdr.create_node_group(
    node_group_name := left_dc,
    parent_group_name := top_group,
    join_node_group := false
);
SELECT bdr.create_node_group(
    node_group_name := right_dc,
    parent_group_name := top_group,
    join_node_group := false
);

- - create a commit scope with individual rules
- - for each sub-group
SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := left_dc,
    rule := ALL (left_dc) AND ANY 1 (right_dc)
);
SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := right_dc,
    rule := ANY 1 (left_dc) AND ALL (right_dc)
);

Confirmation levels

BDR nodes can send confirmations for a transaction at different points in time, similar to Enable Commit At Most Once . In increasing levels of protection, from the perspective of the confirming node, these are:

  • received — A remote BDR node confirms the transaction immediately after receiving it, prior to starting the local application.

  • replicated — Confirm after applying changes of the transaction but before flushing them to disk.

  • durable — Confirm the transaction after all of its changes are flushed to disk.

  • visible (default) — Confirm the transaction after all of its changes are flushed to disk and it’s visible to concurrent transactions.

In rules for commit scopes, you can append these confirmation levels to the node group definition in parenthesis with ON as follows:

  • ANY 2 (right_dc) ON replicated

  • ALL (left_dc) ON visible (default and may as well be omitted)

  • ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable

Reference

Commit scope grammar

For reference, the grammar for commit scopes is composed as follows:

commit_scope:
    confirmation [AND ...]

confirmation:
    node_def (ON [received|replicated|durable|visible])

node_def:
    ANY num (node_group [, ...])
    | MAJORITY (node_group [, ...])
    | ALL (node_group [, ...])

Note

While the grammar for synchronous_standby_names and Commit Scopes looks very similar, it is important to note that the former does not account for the origin node, but the latter does. Therefore, for example synchronous_standby_names = ‘ANY 1 (..)’ is equivalent to a Commit Scope of ANY 2 (…) . This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction.

Adding a commit scope rule

The function bdr.add_commit_scope creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope.

Alternatively, you can invoke it multiple times with the same commit_scope_name but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction.

Synopsis

bdr.add_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME,
    rule TEXT)

Changing a commit scope rule

To change a specific rule for a single origin node group in a commit scope, you can use the function bdr.alter_commit_scope .

Synopsis

bdr.alter_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME,
    rule TEXT)

Removing a commit scope rule

You can use bdr.remove_commit_scope to drop a single rule in a commit scope. If you define multiple rules for the commit scope, you must invoke this function once per rule to fully remove the entire commit scope.

Synopsis

bdr.remove_commit_scope(
    commit_scope_name NAME,
    origin_node_group NAME)

Note

Removing a commit scope that is still used as default by a node group is not allowed