Group Commit

The goal of Group Commit is to provide protection against data loss in case of single node failures or temporary outages. This is achieved by requiring more than one BDR node to successfully receive and confirm a transaction at COMMIT time.

Requirements

During normal operation, Group Commit it completely transparent to the application. Upon failover, the reconciliation phase needs to be explicitly triggered or consolidated by either the application or a proxy in between. HARP provides native support for Group Commit and will trigger the reconciliation phase, making this equally transparent to the client.

On the origin node, a transaction committed with Group Commit uses two-phase commit underneath. Therefore, max_prepared_transactions needs to be configured high enough to handle all such transactions originating per node.

Configuration

To use Group Commit, a Commit Scope first needs to be defined. This determines the BDR nodes involved in the commit of a transaction. Once a scope is established, a transaction may be configured to use Group Commit as follows:

BEGIN;
SET LOCAL bdr.commit_scope = example_scope;
...
COMMIT;

To complete this example, the Commit Scope in question may previously have been defined as:

SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := example_bdr_group,
    rule := ANY 2 (example_bdr_group)
);

This assumes a Node Group named example_bdr_group exists and includes at least two BDR nodes as members (either directly or in sub-groups). Any transaction committed within the example_scope would require one extra confirmation from a BDR node within the group. Together with the origin node, this accounts for “ANY 2” nodes out of the group, on which the transaction is guaranteed to be visible and durable after the commit.

Origin Groups

Rules for Commit Scopes may depend on which node the transaction is committed on, i.e. which node acts as the origin for the transaction. To make this transparent for the application, BDR allows a Commit Scope to define different rules depending on where the transaction originates from.

For example, consider a BDR cluster with nodes spread across two data centers, a left and a right one. Assume the top-level BDR node group is called top_group . The following commands may be used to setup sub-groups and create a Commit Scope requiring all nodes in the local data center to confirm the transaction, but only one node from the remote one.

- - create sub-groups
SELECT bdr.create_node_group(node_group_name := left_dc,
                             parent_group_name := top_group,
                             join_node_group := false);
SELECT bdr.create_node_group(node_group_name := right_dc,
                             parent_group_name := top_group,
                             join_node_group := false);

- - create a commit scope with individual rules for each sub-group
SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := left_dc,
    rule := ALL (left_dc) AND ANY 1 (right_dc)
);
SELECT bdr.add_commit_scope(
    commit_scope_name := example_scope,
    origin_node_group := right_dc,
    rule := ANY 1 (left_dc) AND ALL (right_dc)
);

Confirmation Levels

BDR nodes can send confirmations for a transaction at different points in time, similar to Commit At Most Once . In increasing levels of protection, from the perspective of the confirming node, these are:

  • received - a remote BDR node confirms the transaction immediately after having received it, prior to starting local application

  • replicated - confirm after applying changes of the transaction, but before flushing them to disk

  • durable - confirm the transaction after all of its changes have been flushed to disk

  • visible (default) - confirm the transaction after all of its changes have been flushed to disk and it has been made visible to concurrent transactions.

In rules for Commit Scopes, these confirmation levels may be appended to the node group definition (in parenthesis) with ON as follows:

  • ANY 1 (right_dc) ON remote_write

  • ALL (left_dc) ON remote_commit_flush (default and may as well be omitted)

  • ALL (left_dc) ON remote_commit_async AND ANY 1 (right_dc) ON remote_write

Reference

Commit Scope Grammar

For reference, the grammar for commit scopes is as follows:

commit_scope: confirmation
              | confirmation AND commit_scope

confirmation: node_def (ON [received|replicated|durable|visible])

node_def:     ANY num (node_group [, ...])
              | MAJORITY (node_group [, ...])
              | ALL (node_group [, ...])

Adding a commit scope rule

The function bdr.add_commit_scope creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the BDR cluster, invoking this function once for the top-level node group is sufficient to fully define the commit scope.

Alternatively, it may be invoked multiple times with the same commit_scope_name but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction.

Synopsis

bdr.add_commit_scope(commit_scope_name NAME, origin_node_group NAME,
                     rule TEXT)

Changing a commit scope rule

To change a specific rule for a single origin node group within a commit scope, the function bdr.alter_commit_scope may be used.

Synopsis

bdr.alter_commit_scope(commit_scope_name NAME, origin_node_group NAME,
                       rule TEXT)

Removing a commit scope rule

The bdr.remove_commit_scope can be used to drop a single rule within a commit scope. If multiple rules are defined for the commit scope, this function must be invoked once per rule to fully remove the entire commit scope.

Synopsis

bdr.remove_commit_scope(commit_scope_name NAME, origin_node_group NAME)