Parallel Apply
==============

What is Parallel Apply?
-----------------------

Parallel Apply is a feature of PGD that allows a PGD node to use
multiple writers per subscription. This behavior generally increases the
throughput of a subscription and improves replication performance.

Configuring Parallel Apply
^^^^^^^^^^^^^^^^^^^^^^^^^^

Two variables control Parallel Apply in PGD 5: :ref:`bdr.max_writers_per_subscription <bdr.max_writers_per_subscription>` 

(defaults to 8) and :ref:`bdr.writers_per_subscription <bdr.writers_per_subscription>` 

(defaults to 2).

.. code:: plain

   bdr.max_writers_per_subscription = 8
   bdr.writers_per_subscription = 2

This configuration gives each subscription two writers. However, in some
circumstances, the system might allocate up to eight writers for a
subscription.

You can only change :ref:`bdr.max_writers_per_subscription <bdr.max_writers_per_subscription>` 

with a server restart.

You can change :ref:`bdr.writers_per_subscription <bdr.writers_per_subscription>` 

for a specific subscription without a restart by:

1. Halting the subscription using :ref:`bdr.alter_subscription_disable <bdr.alter_subscription_disable>`  .

2. Setting the new value.

3. Resuming the subscription using :ref:`bdr.alter_subscription_enable <bdr.alter_subscription_enable>`  .

First though, establish the name of the subscription using
``select * from bdr.subscription`` . For this example, the subscription
name is ``bdr_bdrdb_bdrgroup_node2_node1`` .

.. code:: sql

   SELECT bdr.alter_subscription_disable (bdr_bdrdb_bdrgroup_node2_node1);

   UPDATE bdr.subscription
   SET num_writers = 4
   WHERE sub_name = bdr_bdrdb_bdrgroup_node2_node1;

   SELECT bdr.alter_subscription_enable (bdr_bdrdb_bdrgroup_node2_node1);

When to use Parallel Apply
^^^^^^^^^^^^^^^^^^^^^^^^^^

Parallel Apply is always on by default and, for most operations, we
recommend leaving it on.

Monitoring Parallel Apply
^^^^^^^^^^^^^^^^^^^^^^^^^

To support Parallel Apply’s deadlock mitigation, PGD 5.2 adds columns to
 :ref:`bdr.stat_subscription <bdr.stat_subscription>`  . The new columns are ``nprovisional_waits`` ,
``ntuple_waits`` , and ``ncommmit_waits`` . These are metrics that
indicate how well Parallel Apply is managing what previously would have
been deadlocks. They don’t reflect overall system performance.

The ``nprovisional_waits`` value reflects the number of operations on
the same tuples being performed by concurrent apply transactions. These
are provisional waits that aren’t actually waiting yet but could.

If a tuple’s write needs to wait until it can be safely applied, it’s
counted in ``ntuple_waits`` . Fully applied transactions that waited
before being committed are counted in ``ncommit_waits`` .

Disabling Parallel Apply
^^^^^^^^^^^^^^^^^^^^^^^^

To disable Parallel Apply, set :ref:`bdr.writers_per_subscription <bdr.writers_per_subscription>`  to ``1`` .

Deadlock mitigation
^^^^^^^^^^^^^^^^^^^

When Parallel Apply is operating, the transactional changes from the
subscription are written by multiple writers. However, each writer
ensures that the final commit of its transaction doesn’t violate the
commit order as executed on the origin node. If there’s a violation, an
error is generated and the transaction can be rolled back.

This mechanism previously meant that when the following are all true,
the resulting error could manifest as a deadlock:

-  A transaction is pending commit and modifies a row that another
   transaction needs to change.

-  That other transaction executed on the origin node before the pending
   transaction did.

-  The pending transaction took out a lock request.

Additionally, handling the error could increase replication lag, due to
a combination of the time taken:

-  To detect the deadlock

-  For the client to roll back its transaction

-  For indirect garbage collection of the changes that were already
   applied

-  To redo the work

This is where Parallel Apply’s deadlock mitigation, introduced in PGD
5.2, can help. For any transaction, Parallel Apply looks at transactions
already scheduled for any row (tuple) that the current transaction wants
to write. If it finds one, the row is marked as needing to wait until
the other transaction is committed before applying its change to the
row. This approach ensures that rows are written in the correct order.

.. raw:: html

   <!-- ### On-demand dedicated writer

   In PGD 5.4, an on-demand writer mechanism was added to Parallel Apply. This handles the commit phase of prepared GROUP COMMIT transactions by dedicating a writer to participate in the commit confirmation of the GROUP COMMIT. Doing so enables the continued free flow of non GROUP COMMIT transactions through remaining Parallel Apply writers while ensuring consistency for the GROUP COMMIT. -->

Parallel Apply Support
^^^^^^^^^^^^^^^^^^^^^^

Up to and including PGD 5.1, don’t use Parallel Apply with Group Commit,
CAMO, and Eager Replication. Disable Parallel Apply in these scenarios.
If, using PGD 5.1 or earlier, you’re experiencing a large number of
deadlocks, you might also want to disable Parallel Apply or consider
upgrading.

From PGD 5.2, Parallel Apply works with CAMO. It isn’t compatible with
Group Commit or Eager Replication, so disable it if Group Commit or
Eager Replication are in use.