日本語マニュアル
This document introduces the architecture, installation, configuration, and usage of EDB Postgres Replication Server, version 7. EDB Postgres Replication Server (referred to hereafter as EDB Replication Server) is a replication streaming system available for PostgreSQL® and for EDB Postgres™ Advanced Server. The latter will be referred to as Advanced Server.
Note: Direct upgrade from EDB Replication Server 6.x to EDB Replication Server 7 is not supported.
In the following descriptions, a term refers to any word or group of words that are language keywords, user-supplied values, literals, etc. A term’s exact meaning depends upon the context in which it is used.
•
Italic font introduces a new term, typically, in the sentence that defines it for the first time.
•
Fixed-width (mono-spaced) font is used for terms that must be given literally such as SQL commands, specific table and column names used in the examples, programming language keywords, etc. For example, SELECT * FROM emp;
•
Italic fixed-width font is used for terms for which the user must substitute values in actual usage. For example, DELETE FROM table_name;
•
Square brackets [ ] denote that one or none of the enclosed terms may be substituted. For example, [ a | b ] means choose one of “a” or “b” or neither of the two.
•
Braces {} denote that exactly one of the enclosed alternatives must be specified. For example, { a | b } means exactly one of “a” or “b” must be specified.
•
Ellipses ... denote that the preceding term may be repeated. For example, [ a | b ] ... means that you may have the sequence, “b a a b a”.
•
Most of the information in this document applies to both the PostgreSQL and EDB Postgres Advanced Server database systems. The term Advanced Server is used to refer to EDB Postgres Advanced Server. The term Postgres is used to generically refer to both PostgreSQL and Advanced Server. When a distinction needs to be made between these two database systems, the specific names, PostgreSQL or Advanced Server are used.
•
The installation directory path of the PostgreSQL or Advanced Server products is referred to as POSTGRES_INSTALL_HOME. For PostgreSQL Linux installations, this defaults to /opt/PostgreSQL/x.x for version 10 and earlier. For later versions, use the PostgreSQL community packages. For Advanced Server Linux installations accomplished using the interactive installer for version 10 and earlier, this defaults to /opt/edb/asx.x. For Advanced Server Linux installations accomplished using an RPM package, this defaults to /usr/edb/asx.x. The product version number is represented by x.x or by xx for version 10 and later.
-Xmx in the ngxReplicationServer-7.config file located at /usr/edb/rs-7.0/common/etc/sysconfig as shown below. These changes apply while EDB Replication Server is starting.
Note: These source and target versions although supported are not tested and formally certified (in the development environment).
These source and target versions have been tested and formally certified (in the development environment).
Note: These source and target versions although supported are not tested and formally certified (in the development environment).
Note: These source and target versions although supported are not tested and formally certified (in the development environment).
Note: When the ports are open the system is accessible from the outside, which poses a security risk. Make sure to keep only those ports open which are required for EDB Replication Server.
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/monitor/etc
Note: For a two-node cluster there will be
EPRS_HOME/client/etc
Note: EPRS_HOME is the directory where the EDB Replication Server package has been installed. The general format of the EPRS_HOME directory is /usr/edb/rs-x.x where x.x is the EDB Replication Server version number.
Step 1: Check the firewall status:
Step 2: Enable the firewall if it is disabled:
Step 3: Display all the allowed ports:
Step 4: Add a port to the allowed ports to open it for incoming traffic:
Used to set options permanently. Without the --permanent option, a change will only be a part of the runtime configuration.
Step 4: Reload FirewallD configuration after using the --permanent argument to make changes to your firewall configuration.
Step 5: Check if the ports are added to the allowed list:
By default iptables firewall stores its configuration at /etc/sysconfig/iptables file on RHEL/CentOS 6.x. Edit this file and add rules to open port number.
Step 1: Open the file /etc/sysconfig/iptables:
Step 2: Add the following rule:
--dport portid -j ACCEPT
Step 3: Save and close the file. Restart iptables:
Step 1: Execute the iptables command:
Step 2: Rules created with the iptables command are stored in the memory. If the system reboots before you save the iptables rule set, all the rules are lost. To save the rules permanently run the following command as root:
Step 3: To verify if the ports have been opened run the following command:
Example: Open port 2181 and 2182 on CentOS 6
•
Section 2.2.1 describes the open source components providing the EDB Replication Server infrastructure. Links to various websites are given, which contain complete, detailed information of these open source products.
•
Section 2.3 describes the EDB Replication Server components that you create and manage to engage the open source components for performing the data streaming functionality.
•
Section 2.4 shows some basic examples of how the EDB Replication Server components and the main open source components engage in a replication network supporting an SMR and MMR cluster.
Apache Kafka is the primary open source component providing the data streaming functionality used by EDB Replication Server. Kafka and the other key components are the following:
•
Apache Kafka. An open-source distributed streaming platform developed by the Apache Software Foundation. Kafka is a messaging system for publishing and subscribing to streams of records in real-time.
•
Apache ZooKeeper. A centralized service for maintaining configuration information, distributed synchronization, and group services for the distributed components of Kafka.
•
REST or RESTful. Representational State Transfer is an architectural style that facilitates the implementation of networked and web-enabled applications.
•
Node. A single computer running one or more Kafka brokers.
•
Broker. A Kafka process instance maintaining the published data on a node.
•
Cluster. A group of multiple nodes or brokers. A cluster is used to manage the persistence and replication of published message data.
•
Topic. A category identifying a particular stream of messages to be published.
•
Partition. Each topic can be divided into a number of partitions. Each partition contains messages. Each message within its partition is uniquely identified by its sequence offset. Partitions for a given topic may be managed by separate brokers.
•
Producer. An application that publishes messages to one or more Kafka topics. The message is sent to a Kafka broker.
•
Consumer. An application that reads data from Kafka brokers. A consumer subscribes to one or more topics and pulls data from the brokers.
•
Replication Server. A replication server is a process running as a dedicated RESTful service on an HTTP server that engages a single Kafka broker and either one or two ZooKeeper instances. Each replication server can have none, one, or more associated databases that act as a producer or consumer for Kafka. The replication server is the first EDB Replication Server component that must be created and started when creating a replication node.
•
Leader Service. The replication server used as the primary replication server in the replication network. The leader service is the replication server with which the user issued RepCLI commands communicate. The first replication server started when creating a new replication network acts as the leader service and has two ZooKeeper instances running in its framework. All other additional replication servers added to the replication network have one ZooKeeper instance. During the course of operation of the replication network, the leader service role may be transferred to another replication server resulting from a failover operation (that is, the current replication server acting as the leader service aborts so the leader service operation is changed to another active replication server in order to keep the replication network active).
•
Replication Node. A combination of a single replication server, its underlying Kafka broker and ZooKeeper instances, any producer and consumer databases that have been added to that replication server, publications created within the replication server, topic partitions within the Kafka broker resulting from the publication creations, as well as all other supporting open source components. A replication node may have no producer or consumer databases added to its replication server in order to serve the purpose of enhancing high availability support. In this situation, if another replication server aborts, its functionality can be transferred to this other replication server. A replication node forms a single, complete EDB Replication Server component that can contribute to a full replication network.
•
Replication Network. A group of one or more replication nodes. Each replication node must be registered to the replication network, which is accomplished by joining its replication server to the replication network. The first replication server started and joined to the replication network starts as the leader service. All of the
•
Publication. A defined set of one or more tables from a given producer database whose changed data is streamed to consumer databases. A publication is implemented as Kafka topics for storing the changed data. The changed data is streamed to consumer databases that have been registered as part of the replication network and have been joined to that particular publication. A publication is identified by a name that must be unique amongst all publications in the replication network.
•
Replication Command Line Interface (RepCLI). The command line tool to perform the setup, configuration, and execution process for EDB Replication Server. The RepCLI commands are supported by the RESTful architecture.
Chapter 5 presents examples of how the replication networks described in this section are created.
/var/lib/edb/rs/data
When changed data occurs within a publication table, the changes are collected using PostgreSQL logical decoding. The changed data is extracted from the PostgreSQL Write-Ahead Log segments (WAL segment files). This information is then converted and stored in the Kafka topics managed by the Kafka broker. There may be a multiple number of topics managing a publication.
Though Kafka supports the use of topics divided into multiple log files called partitions, the EDB Replication Server uses only one partition per topic.
servername is the identifier assigned to the replication server when it has been joined to the network.
Section 6.1.1 provides an example of configuring this SMR cluster.
Section 6.1.2 provides an example of configuring these two replication node SMR cluster.
Section 6.2.1 provides an example of configuring this MMR cluster.
Section 6.2.2 provides the configuration example of how this is accomplished.
The edb-rs package is installed using a repository RPM, which contains URLs to access the EDB Yum Repository for various components or from the edb-rs RPM package files directly downloadable from an EnterpriseDB website location to be determined. The repository RPM is shown when you access the EDB Yum Repository website.
For information about using the EDB Yum Repository see Chapter 3 of the EDB Postgres Advanced Server Installation Guide available from the EnterpriseDB website at:
Each EDB Replication Server component is available as an individual RPM package. Thus, you can install all components with a single yum install command, or you may choose to install selected, individual components by installing only those particular RPM packages.
The Advanced Server libs package must be available for access by Yum when installing any EDB Replication Server RPM package component. The edb-as10-server-libs package is a component of the Advanced Server repository package for version 10 that must be installed. Step 3 shows how to enable access to the Advanced Server repository so Yum can access its server libs package.
yum install package_name
package_name is any of the packages listed under the Package Name column of the preceding table.
Step 1: You must have Java Runtime Environment (JRE) version 1.8 installed on the hosts where you intend to install any EDB Replication Server component. Any Java product such as Oracle Java or OpenJDK may be used.
Step 2: From the EDB Yum Repository, click on the edb-repo link to download the repository RPM for all EnterpriseDB RPMs.
As the root account, issue the following command to install this repository configuration package:
Step 3: In directory /etc/yum.repos.d, the repository configuration file edb.repo is created, which is text file containing a list of EnterpriseDB repositories, each denoted by an entry starting with the text [repository_name].
•
Repository edbasx for Advanced Server
x is the Advanced Server version
•
Repository enterprisedb-dependencies
•
Repository enterprisedb-tools
Step 4: Install the EDB Replication Server RPM package.
The EDB Replication Server is installed in directory location /usr/edb/rs-x.x where x.x is the product version number as shown by the following:
•
Client Application. Command line based application for users to interact with the server to perform various operations
•
Server Application. Command line application for starting the replication server and for handling the various replication server services such as the Kafka broker, ZooKeeper, and the Schema Registry
•
Monitoring. Command line based application for monitoring the replication server components
•
Data Validator. Command line based application for comparing and validating the rows of pairs of tables
•
Common. JAR files and scripts shared in common by the applications
EPRS_HOME/client/bin
EPRS_HOME/client/etc
EPRS_HOME/client/etc
EPRS_HOME/common/etc
EPRS_HOME/common/etc/sysconfig
EPRS_HOME/server/bin
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/server/etc
EPRS_HOME/monitor/bin
EPRS_HOME/monitor/etc
EPRS_HOME/monitor/etc
EPRS_HOME/datavalidator/bin
EPRS_HOME/datavalidator/etc
EPRS_HOME is the directory where the EDB Replication Server package has been installed. The general format of the EPRS_HOME directory is /usr/edb/rs-x.x where x.x is the EDB Replication Server version number, which is initially 7.0.
•
/var/lib/edb/rs/data contains the data and other components created and used by the replication network such as ZooKeeper and topics.
•
/var/log/edb/rs contains the log files.
You can uninstall any component by invoking the yum remove package_name command as the root account where package_name is any component RPM package as listed in the table in Section 3.1.
•
The database tables to be used in publications must exist in the producer and consumer databases before creating the replication network. The table definitions must exist (that is, the structure created by the CREATE TABLE command), but the tables can initially be empty. The consumer database tables can be loaded with the rows from the producer database using the startsnapshot command once the replication network has been created.
•
The database tables referred to in the previous bullet point must have REPLICA IDENTITY FULL defined on the tables. This can be done with the SQL command ALTER TABLE schema.table REPLICA IDENTITY FULL.
•
An encrypted form of the database user password must be specified when adding a database to the replication network. Use the encrypt RepCLI command to generate the encrypted form of the password into an output text file.
The following is the encrypt command where the unencrypted form of the database user password is given in unencrypted_password_file:
./runRepCLI.sh –encrypt –input unencrypted_password_file
–output encrypted_password_file -user username
Note: The following permission is required for encrypt command:
The runRepCLI.sh script is run from the EPRS_HOME/client/bin directory.
2.
Join the replication server to the replication network. The first replication server that is joined to the network becomes the replication server acting as the leader service of the replication network. There is one and only one leader service per replication network.

This first command to join the network as the leader service as well as all subsequent commands for adding other replication servers to this replication network and to perform all of the following processes in this list must be run from the EPRS_HOME/client/bin directory located on the host running the leader service.

During this step, ZooKeeper is started up. Two instances of ZooKeeper are started on the leader service. A single ZooKeeper instance is started on each additional replication server that is joined to the network.

In addition, after starting the leader service, an administrative user named admin is created. You must then set the password for the admin user. See Section 4.2.3 for setting the admin user password. See Section 4.2.2 for joining the network.
•
When setting up within a given host. Step 1 (starting the replication server) and Step 2 (joining the network) must be done first. Step 3 (adding a database) can then be followed by Step 4 (creating a publication). For the consumers, Step 3 (adding a database) is repeated followed by Step 5 (joining a publication). Alternatively, Step 3 (adding a database) can be done several times for multiple databases before proceeding with Step 4 (creating a publication) and Step 5 (joining a publication) on consumer databases that have been added.
•
When setting up more than one host for the replication network. Step 1 (starting the replication server) and Step 2 (joining the network) can be done on several hosts before performing the remaining steps on any host on which steps 1 and 2 have been done.
As the root account, run the following from the EPRS_HOME/server/bin directory:
./runServer.sh --host host_ip_address
--config EPRS_HOME/server/etc
-h, --host host_ip_address
-c, --config EPRS_HOME/server/etc
The full directory path to the EPRS_HOME/server/etc directory. If you want to have multiple replication servers running on the same host machine, you need to make a copy of the EPRS_HOME/server directory and modify certain properties files located in the EPRS_HOME/server/etc directory of the additional replication server. See the EDB Postgres Replication Server Reference Guide for information on those properties files that must be modified.
Generally, all subsequent RepCLI commands must be run from the EPRS_HOME/client/bin directory on the host running the replication server leader service in order for the effect of the RepCLI commands to be applied to that replication network.
Alternatively, if the client package has been installed on a different host than where the leader service is running, you can run the RepCLI commands from this separate host if the client.properties file in the EPRS_HOME/client/etc directory is edited to specify the host IP address and port of the leader service.
Run the following from the EPRS_HOME/client/bin directory:
-host host_ip_address -port port [ -ngxpasspath directory ]
[ -user username ]
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Replication user running the command. This parameter is not required when running the joinnetwork command for the first time in a replication network when creating the leader service but is required for all subsequent usage of joinnetwork and all other RepCLI commands.
After the first joinnetwork command has been run establishing the leader service, additional replication servers are added to the network by running the joinnetwork command specifying a unique servername to identify the replication server and host_ip_address to specify the host location where this additional replication server is running.
The -port 8082 option is the default port number assigned to the replication server as given by the ngen.server.port parameter in the application.properties file. If this port number was modified in order to run multiple replication servers on the same host, the -port option must specify the altered port number.
When joinnetwork is used to establish the leader service, an admin user is created with the administrator role, which has all permissions. The password for the admin user must then be set with the -setadminpassword command as in the following:
A prompt then appears for the admin user password. The password must then be entered for all subsequent RepCLI commands when -user admin is specified stating that admin is the user running the command unless the -savepassword option had been included.
The database server running the database must have been configured to support its access by a replication server. For databases that are to be used as producers, see Section 4.2.19 for configuration information. For databases that are to be used as consumers, see Section 4.4 for configuration information.
Run the following from the EPRS_HOME/client/bin directory:
./runRepCLI.sh -adddb [ -servername servername ] -dbid dbid
-dbport port -dbuser user
{ -dbpassword encrypted_pwd | -dbpassfile pwdfile }
-database dbname [ -ngxpasspath directory ] -user username
Specify postgresql if the database is a PostgreSQL database or enterprisedb if it is an Advanced Server database.
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Run the following from the EPRS_HOME/client/bin directory:
-servername servername
-dbid dbid
{ -tables schema_1.table_1[,schema_2.table_2 ]... |
-alltables [ schema_1][,schema_2 ]... }
-user username
The name of the replication server to which the publication is to be added. The database specified by dbid, containing the tables of the publication must also have been added to this replication server.
schema_n.table_n
-alltables [ schema_1][,schema_2 ]...
Specify W if the database can only write to this publication, which makes it a producer only. Specify RW if the database can also accept changed data from other databases that have joined this publication. The latter makes this database both a consumer and producer of the publication. If -nodetype is omitted, the default is W.
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Note: All the tables to be replicated must have a primary key otherwise, it will throw an error while creating a publication.
Run the following from the EPRS_HOME/client/bin directory:
-pubname pubname [ –nodetype { R | W | RW } ]
[ -filtername filtername ]
[ -ngxpasspath directory ] -user username
Specify R if the database can only read from this publication, which makes it a consumer only. Specify W if the database can only write to this publication, which makes it a producer only. Specify RW if the database can stream and receive changed data to and from other databases that have joined this publication. The latter makes this database both a consumer and producer of the publication. If -nodetype is omitted, the default is R.
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Note that typically, the joinpub command is used when creating the initial replication network before snapshots are taken to all target databases and streaming has started.
•
•
•
Rerun the startstreaming command.
Note that the startstreaming command must be re-executed even if streaming is already running on the replication network.
Run the following command from the EPRS_HOME/client/bin directory:
{-tables table_1, table_2 ...} [ -ngxpasspath directory ]
–user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
•
To use the addtables command you should have the create_pub permission.

Run the following command from the EPRS_HOME/client/bin directory:
{-tables table_1, table_2 ...} [ -ngxpasspath directory ]
–user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
•
To use the removetables command you should have the remove_pub permission.
Publication pubname has no table.
Note: To check the tables in the specified publication use the listpubtable command.
Note (For MMR only): When using table filters in a multi-master replication system, the master definition node, which provides the source of the table content for a snapshot, should contain a superset of all the data contained in the other master nodes of the multi-master replication system. This ensures that the target of a snapshot receives all of the data that satisfies any filtering criteria enabled on the other master nodes.
Note: In the following discussion, a result set refers to the set of rows in a table satisfying the selection criteria of an UPDATE or DELETE statement executed on that table.
When an INSERT statement is executed on a source table followed by a synchronization replication, the row is inserted into the target table of the synchronization if the row satisfies the filtering criteria. Otherwise, the row is excluded from insertion into the target table.
When an UPDATE statement is executed on a source table followed by a synchronization replication, the UPDATE result set of the source table determines the action on the target table of the synchronization as follows.
When a DELETE statement is executed on a source table followed by a synchronization replication, the DELETE result set of the source table determines the action on the target table of the synchronization as follows.
Thus, regardless of whether the transaction on the source table is an INSERT, UPDATE, or DELETE statement, the goal of a table filter is to ensure that all rows in the target table satisfy the filter rule.
The REPLICA IDENTITY FULL setting is required on tables in the following databases of a log-based replication system:
•
In a multi-master replication system, non-MDN nodes should not have their tables’ REPLICA IDENTITY option set to FULL unless transactions are expected to be targeted on those non-MDN nodes, and the transactions are to be filtered when they are replicated to the other master nodes.
The REPLICA IDENTITY FULL setting on a source table ensures that certain types of transactions on the source table result in the proper updates to the target tables on which filters have been enabled.
This setting is done with the ALTER TABLE command as shown by the following:
ALTER TABLE schema.table_name REPLICA IDENTITY FULL
For example, for a publication table named edb.dept, use the following ALTER TABLE command:
The REPLICA IDENTITY setting can be displayed by the PSQL utility using the \d+ command:
For additional information see the ALTER TABLE SQL command in the PostgreSQL Core Documentation located at:
•
•
•
•
•
Note: The following permission is required for taking the snapshot:
Start a snapshot with startsnapshot.
Run the following from the EPRS_HOME/client/bin directory:
-dbid target_dbid [ -ngxpasspath directory ] –user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
The checksnapshot command confirms if the data is replicated to the target node.
Run the following from the EPRS_HOME/client/bin directory:
-dbid target_dbid
[ -ngxpasspath directory ]
–user username
Now run checksnapshot command after you run the startSnapshot command (after a delay of few seconds) to confirm if data is replicated to target node. The status should be completed (for Data Publishing as well as Data Import).
Note: The following permissions are required for taking and checking the snapshot status:
Start a snapshot with startsnapshot.
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
If you change the cluster configuration (remove a database, publication, and/or subscription) after a snapshot operation, repeat the snapshot by including the reload option. This is necessary as the Kafka queues (topics) are re-populated with a fresh copy of data from the source database.
Stop streaming before you execute the startsnapshot command with the reload option, otherwise, the operation will fail (an error message is logged in the server). Once the snapshot is completed explicitly restart the streaming.
Note: The reload option does not work for an offline snapshot.
Run the following command from the EPRS_HOME/client/bin directory:
-dbid target_dbid
[ -ngxpasspath directory ]
–user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
The following is a sample execution of the startsnapshot command with the reload option.
Note: Filters do not work for an offline snapshot as the data movement takes place outside the replication server.
Step 1 (Optional): If you already have the published tables and schemas on the target database omit this step. Make sure that schema definition for all the publication tables is proper on the target database.
Run the following command from the /opt/edb/database_server_installation_directory/bin directory:
./pg_dump -U postgres -p port -Ft -s source_database_name > source_database_name_schemaonly.sql.tar
Note: If partial set of tables are a part of the publication, take a partial backup (for selected tables with the –tables option).
In this example the following command is run from the /opt/edb/as10/bin directory where Advanced Server 10 is running on port 5432.
$ ./pg_dump -U postgres -p 5432 -Ft -s db1 > db1_schemaonly.sql.tar
Step 2 (Optional): If you already have the published tables and schemas on the target database omit this step. Make sure that the schema definition for all the publication tables is proper on the target database.
To restore the dump on the target database you can use any method that is convenient for you. Here, the pg_restore utility is used to restore the dump on the target PostgreSQL database as follows:
./pg_restore -U postgres -p port -d target_database_name source_database_name_schemaonly.sql.tar
Step 3: Create a publication.
Step 4: Join a publication on the target node.
Step 5: Take the snapshot (specify the offline option).
Run the following from the EPRS_HOME/client/bin directory:
-dbid target_dbid [ -ngxpasspath directory ] –user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Step 6: Run the checksnapshot command and note down the exported Snapshot Id.
Step 7: Take a backup of the publication database (data only) using the exported Snapshot Id. Skip control schema _ngx_rep_cluster from the backup.
Run the following command from the /opt/edb/ database_server_installation_directory/bin directory after replacing the Snapshot_Id with the exported Snapshot Id from step 6:
./pg_dump -U postgres -p port -Ft -a --snapshot=Snapshot_Id --exclude-schema=_ngx_rep_cluster source_database_name > source_database_name_dataonly.sql.tar
Note: The control schema name is configurable. The default control schema name is _ngx_rep_cluster.
Step 8: Restore the database dump (data only) on the subscription database. To restore the dump on the target database you can use any method that is convenient for you. Here, the pg_restore utility is used to restore the dump on the target PostgreSQL database as follows:
./pg_restore -U postgres -p port -d target_database_name source_database_name_dataonly.sql.tar
Step 9: Start streaming for the target database.
Run the following from the EPRS_HOME/client/bin directory:
./runRepCLI.sh –startstreaming -pubname pubname [-dbid target_dbid ][ -ngxpasspath directory ] -user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
./runRepCLI.sh –stopstreaming -pubname pubname [ -dbid target_dbid ][ -ngxpasspath directory ] -user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
As shown in Section 4.2.1 a replication server is started from a terminal.
•
In a separate terminal as the root user, issue the command lsof -i tcp:8082 to return the process ID of the replication server, then run the command kill -9 process_id to stop the replication server. In the lsof command, 8082 is the port used by the replication server as given by the ngen.server.port parameter in the EPRS_HOME/server/etc/application.properties file.
When the host machine is restarted, start the replication server as shown in Section 4.2.1 to have the replication node rejoin the replication network. The other RepCLI commands that have already been used to configure the node do not have to be repeated.
•
leavepub removes a database from a publication to which it had been joined to with the joinpub command.
•
removepub deletes a publication from the replication server in which it had been created with the createpub command.
•
removedb removes a database from a replication server that had been added with the adddb command.
•
leavenetwork removes a replication server from the replication network that had been added with the joinnetwork command.
See the EDB Postgres Replication Server Reference Guide for information about these commands.
To enable adding events in Kafka uncomment the following line in EPRS_HOME/server/etc/logback.xml before you start the replication servers.
Run the following command from the EPRS_HOME/client/bin directory:
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
The following is an example of listevents command with MMR (two servers) setup:
Run the following command from the EPRS_HOME/client/bin directory:
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Note: For receiving all the messages uncomment the following line in EPRS_HOME/server/etc/logback.xml file.
The following is an example of filtering listevents command with severity as error:
Note: If the producer database is also to be a consumer, see Section 4.4 for consumer setup requirements that need to be added as well.
•
wal_level. Set to logical.
•
max_wal_senders. Specifies the maximum number of concurrent connections (that is, the maximum number of simultaneously running WAL sender processes). Should be set to a value greater than the total number of producer databases on the database server.
•
max_replication_slots. Specifies the maximum number of replication slots. Should be set to a value greater than the total number of producer databases on the database server.
•
A Postgres database server uses the host-based authentication file pg_hba.conf to control access to the databases in the database server.
You need to modify the pg_hba.conf file on each Postgres database server that contains a producer database.
host pub_dbname pub_dbuser pub_ipaddr/32
The value you substitute for pub_dbname is the name of the Postgres producer database you want to use (specified with the -database option of the -adddb RepCLI command). The value you substitute for pub_dbuser is the database user name to be specified with the -dbuser option of the adddb RepCLI command. The value you substitute for pub_ipaddr is the host IP address of the replication server that will connect to the database.
The pg_hba.conf file must contain an additional entry with the DATABASE field set to replication for pub_dbname, pub_dbuser, and pub_ipaddr to allow replication connections from the replication server on the host on which it is running.
For a Postgres producer database node1, connecting with the database user enterprisedb, run the following command to add a database:
$EPRS_HOME/client/bin/runRepCLI.sh -adddb -servername rep_server_1 -dbid db1 -dbtype enterprisedb -dbhost 127.0.0.1
-dbport 5444 -dbuser enterprisedb -dbpassword ygJ9AxoJEX854elcVIJPTw== -database node1 -user admin
The resulting pg_hba.conf file appears as follows:
where 127.0.0.1 is the address of the rep_server_1 in the above -adddb command.
node1 is the dbname specified with the -database option of the -adddb RepCLI command.
where 192.168.2.27 is the address of the rep_server_1, specified in the above -adddb command.
node1 is the dbname specified with the -database option of the -adddb RepCLI command.
The database user specified with the -dbuser option of the adddb command must have the following privileges:
•
REPLICATION privilege if the database user is not a superuser
In the pg_hba.conf file, this database user must be included as shown by the pub_dbuser variable in Section 4.3.1.2.
Note: If the consumer database is also to be a producer, see Section 4.3 for producer setup requirements that need to be added as well.
A Postgres database server uses the host-based authentication file pg_hba.conf to control access to the databases in the database server.
You need to modify the pg_hba.conf file on each Postgres database server that contains a consumer database.
host sub_dbname sub_dbuser sub_ipaddr/32
The value you substitute for sub_dbname is the name of the Postgres consumer database you intend to use. The value you substitute for sub_dbuser is the database user name to be specified with the -dbuser option of the adddb RepCLI command. The value you substitute for sub_ipaddr is the host IP address of the replication server that will connect to the database.
For a Postgres Consumer database node2, connecting with the database user enterprisedb, run the following command to add database node 2:
The resulting pg_hba.conf file appears as follows:
where 127.0.0.1 is the address of the rep_server_2 in the above -adddb command.
node2 is the dbname specified with the -database option of the -adddb RepCLI command.
where 192.168.2.28 is the address of the rep_server_2, specified in the adddb command.
node2 is the dbname specified with the -database option of the -adddb RepCLI command.
The database user specified with the -dbuser option of the adddb command must have the following privileges:
Note: Database server user should have superuser privileges in order to perform certain operations such as disablement of constraints on the consumer database tables.
In the pg_hba.conf file, this database user must be included as shown by the sub_dbuser variable in Section 4.4.1.1.
./runRepCLI.sh -replicationlatency -pubname pubname
-dbid target_dbid [ -ngxpasspath directory ]
-user username
The directory containing the .ngxpass password file. If the option is omitted, the .ngxpass file will be searched for in the user’s home directory or by the setting of environment variable NGXPASSPATH.
Note: The following permission is required for replicationlatency command:
•
Single-Master Replication (SMR) Cluster. Database nodes must act as either the producer or consumer of changed data.
•
Multi-Master Replication (MMR) Cluster. Database nodes can act as both producers and consumers of changed data.
Note: Be sure the database objects have been created in the producer and consumer databases as described in Section 4.1.1.
The publication database pubdb initially contains the following rows in the dept table:
The dept table in the consumer database subdb contains no rows.
Step 1: As the Linux root user account, start the replication server on the host to be started as the leader service by invoking the runServer.sh script from the EPRS_HOME/server/bin directory. The IP address of the host on which this script is being run must be specified with the --host option.
Step 2: Open a second terminal on the same host using any Linux user account that has read and execute permissions on the EPRS_HOME/client/bin directory. From the EPRS_HOME/client/bin directory, join the replication network by invoking the runRepCLI.sh script with the joinnetwork command.
When using the joinnetwork command, you specify the IP address of the host running the replication server that is to become a member of the replication network along with the replication server’s port number. In this case, the replication server is running on the host you have logged into in Step 1 and the port number is the replication server’s default setting of 8082.
The first usage of the joinnetwork command on the host on which the replication server has been started results in its usage as the leader service assigning the name given by the -servername option, which is localService in this example.
Step 3: You must now use the setadminpassword command to set the administrator password for the EDB Replication Server administrative user that is given the username of admin. The admin user has the authority to run any RepCLI command. The response you enter for the Enter admin password prompt becomes the administrator password for the admin user.
For the remaining RepCLI commands in these examples, -user admin is given to run the command as the administrative user. There is a set of RepCLI commands that give you the ability to create other usernames and assign them certain roles and permissions for what the user can accomplish. See the EDB Postgres Replication Server Reference Guide for this information.
Step 4: Use the adddb command to add the publication database to the leader service. The replication server name localService, which was assigned in Step 2 is given with the -servername option.
Use the -dbid option to assign a unique database identifier to this database. This database identifier must then be specified in other RepCLI commands. The database identifier db1 is given for this example.
Also note that the -user admin option is given as described in Step 3.
Step 5: Use the createpub command to create the publication to specify the tables whose changed data is to be captured and streamed to other consumer databases.
The -pubname option specifies the publication name you assign to the publication.
The -dbid option specifies the database that contains the tables that are to be in this publication.
The -servername option specifies that this publication is to be part of this replication server, localService.
The -tables option lists one or more tables with their schema that are to be part of the publication.
A multiple list of schema.table_name must be separated by commas with no spaces before or after the comma.
Step 6: Add the subscription database to the replication server to act as the consumer.
The options of the adddb command are described in Step 4.
Note that db2 is the identifier assigned to this database with the -dbid option, which will be referenced by the joinpub command described in the next step.
Step 7: Join the publication so the subscription database can receive the changed data from the publication tables. This is done by using the joinpub command where the -pubname option specifies the publication from which the changed data is to be received.
The -dbid option identifies the subscription database that is to receive this changed data.
The -servername option specifies the replication server to which this subscription database has been added in Step 6.
Note that by default, the joinpub command only allows the database to read changed data from the publication database. Changes that may occur on the subscription database tables are not streamed to other consumers unless the -nodetype RW option is specified with the joinpub command.
Step 8: Perform a snapshot to load these subscription database tables with the rows that are present in the publication database tables.
The -pubname option of the startsnapshot command specifies the publication name.
The -dbid option specifies the target consumer database to receive the snapshot.
Step 9: Run checksnapshot command (after a few seconds) to confirm if the data is replicated to the target node.
Step 10: After the row content of all publication tables is consistent across the publication and subscription databases, start the streaming.
$ ./runRepCLI.sh -startstreaming -pubname deptpub -user admin
Note: Be sure the database objects have been created in the producer and consumer databases as described in Section 4.1.1.
The publication database node1 initially contains the following rows in the dept table:
The dept table in the consumer database node2 contains no rows.
Step 1: Start the replication server on the host to be started as the leader service by invoking the runServer.sh script from the EPRS_HOME/server/bin directory.
Step 2: Open a second terminal on the same host. From the EPRS_HOME/client/bin directory, join the replication network by invoking the runRepCLI.sh script with the joinnetwork command.
Step 3: Set the administrator password for the admin user. Use this password when prompted for it when invoking the subsequent RepCLI commands with the -user admin option.
Step 4: Add the publication database to the leader service. The replication server name that was specified in Step 2 is given with the -servername option.
Use the -dbid option to assign a unique database identifier to this database. This database identifier must then be specified in other RepCLI commands.
Step 5: Create the publication to specify the tables whose changed data is to be captured and streamed to the subscription database.
Step 6: Log into the host machine that is to run the subscriber and invoke the runServer.sh script from the EPRS_HOME/server/bin directory installed on that host.
Step 7: Join the replication server on the subscription host machine to the replication network by running the joinnetwork command specifying the host IP address of the subscription host with the -host option.
Note: Be sure you run the joinnetwork command and all additional RepCLI commands from the leader service host machine and not from the subscriber host you logged into for Step 6.
Step 8: Add the subscription database to the replication server on the subscription host by specifying the replication server name remoteService with the -servername option. The specified replication server in the subsequent joinpub command becomes the source to which the connection to the database is made.
Use the -dbid option to assign a unique database identifier for the database. The database identifier is then specified in other RepCLI commands.
Step 9: Join the publication so the subscription database can receive the changed data from the publication tables. The joinpub command with the -pubname option specifies the publication from which the changed data is to be received.
The -dbid option identifies the subscription database that is to receive this changed data.
The -servername option specifies the replication server from which the database connection is made to receive the replication.
Step 10: Perform a snapshot to load the subscription database tables with the rows that are present in the publication database tables.
The -pubname option of the startsnapshot command specifies the publication name.
The -dbid option specifies the target consumer database to receive the snapshot.
Step 11: Run checksnapshot command (after a few seconds) to confirm if the data is replicated to the target node.
Step 12: After the row content of all publication tables is consistent across the publication and subscription databases, start the streaming.
Note: Be sure the database objects have been created in the producer and consumer databases as described in Section 4.1.1.
The publication database node1 initially contains the following rows in its tables:
The database node2 contains no rows in any of its tables.
Step 1: Start the replication server on the host to be started as the leader service by invoking the runServer.sh script from the EPRS_HOME/server/bin directory.
Step 2: Open a second terminal on the same host. From the EPRS_HOME/client/bin directory, join the replication network by invoking the runRepCLI.sh script with the joinnetwork command.
Step 3: Set the administrator password for the admin user. Use this password when prompted for it when invoking the subsequent RepCLI commands with the -user admin option.
Step 4: Add the publication database to the leader service. The replication server name that was specified in Step 2 is given with the -servername option.
Use the -dbid option to assign a unique database identifier to this database. This database identifier must then be specified in other RepCLI commands.
Step 5: Create the first publication.
The use of the -nodetype RW option, which is read/write on the publication, specifies that the database that owns the tables of the publication, identified by the -dbid option, is to accept and apply changed data to its own tables in addition to streaming its own changes to other consumer databases.
The use of the -nodetype RW option on the createpub command as well as on the joinpub command is what results in an MMR cluster where publication table changes on any database are replicated and applied to all other databases in the cluster.
For the createpub command, if the -nodetype option is omitted, the default effect is -nodetype W, which is write-only, meaning that the database owning the publication tables can have its changes streamed to other database consumers, but it will not accept and apply changes to its own tables from other databases.
Step 6: Create the second publication with the -nodetype RW option.
Step 7: Log into the host machine that is to serve as the second replication node and invoke the runServer.sh script from the EPRS_HOME/server/bin directory installed on that host.
Step 8: Join the replication server on the second replication node host machine to the replication network by running the joinnetwork command specifying the host IP address of the second replication node host with the -host option.
Note: Be sure you run the joinnetwork command from the leader service host machine and not from the second replication node host you logged into for Step 7.
Step 9: Add the other database of the MMR cluster to the replication server on the second replication node host by specifying the replication server name remoteService with the -servername option.
Use the -dbid option to assign a unique database identifier for the database. The database identifier is then specified in other RepCLI commands.
Step 10: Join the first publication so this second MMR database can receive the changed data from the publication tables. The joinpub command with the -pubname option specifies the publication from which the changed data is to be received.
The -dbid option identifies the database that is to receive this changed data.
The -servername option specifies the replication server to which this database has been added in Step 9.
Note that by default, the joinpub command only allows the database to read changed data from the publication database. Changes that may occur on these database tables are not streamed to other consumers unless the -nodetype RW option is specified with the joinpub command.
Step 11: Join the second publication.
Step 12: Perform snapshots to load these database tables with the rows that are present in the publication database tables.
The -pubname option of the startsnapshot command specifies the publication name.
The -dbid option specifies the target database to receive the snapshot.
Step 13: Run checksnapshot command (after a delay of a few seconds) to confirm if the data is replicated on the target node.
Step 14: Perform a snapshot on the second publication.
Step 15: Run checksnapshot command (after a delay of a few seconds) to confirm if the data is replicated to the target node.
Step 16: After the row content of all publication tables is consistent across the databases, start the streaming.
The -pubname option of the startstreaming command specifies the publication name.
Step 17: Start the streaming on the second publication.
Note: Be sure the database objects have been created in the producer and consumer databases as described in Section 4.1.1.
The publication database node1 initially contains the following rows in the dept table:
The other databases, node2 and node3, contain no rows in the dept table.
Step 1: Start the replication server on all nodes.
Log into the host to start as the leader service and invoke the runServer.sh script from the EPRS_HOME/server/bin directory installed on that host:
Log into the host that will run the second replication server and invoke the runServer.sh script from the EPRS_HOME/server/bin directory installed on that host:
Log into the third host and invoke the runServer.sh script from the EPRS_HOME/server/bin directory installed on that host:
All of the subsequent RepCLI commands in the remaining steps must be executed from the EPRS_HOME/client/bin directory installed on the host where usage as the leader service will be established. The RepCLI commands must not be executed from any of the other two host machines.
Step 2: Join all replication servers to the replication network.
From the EPRS_HOME/client/bin directory, join the replication network by invoking the runRepCLI.sh script with the joinnetwork command.
The first joinnetwork command you run establishes its usage as the leader service:
Join the second replication server to the network by running the joinnetwork command specifying the IP address of the host running this replication server with the -host option.
Join the third replication server to the network by running the joinnetwork command specifying the IP address of the host running this replication server with the -host option.
Step 3: Add databases to all replication servers. The -servername option of the adddb command specifies the replication server on the host to which the database is to be added.
Use the -dbid option to assign a unique database identifier for each database. The database identifier is then specified in other RepCLI commands.
Step 4: Create the publication on the leader service with the -nodetype RW option to enable acceptance of changed data from other producers as well as streaming its own changed data to other consumers that have joined the publication.
Step 5: Join the publication so the database can receive the changed data from the publication tables. The joinpub command with the -pubname option specifies the publication from which the changed data is to be received.
The -dbid option identifies the database that is to receive this changed data.
The -servername option specifies the replication server to which the database has been added.
Note that by default, the joinpub command only allows the database to read changed data from the publication database. Changes that may occur on these database tables are not streamed to other consumers unless the -nodetype RW option is specified with the joinpub command.
Step 6: Perform snapshots to load these consumer database tables with the rows that are present in the publication database table.
The -pubname option of the startsnapshot command specifies the publication name.
The -dbid option specifies the target database to receive the snapshot.
Step 7: Run checksnapshot command (after a delay of a few seconds) to confirm if the data is replicated to the target node.
Step 8: Perform snapshots to load these consumer database tables with the rows that are present in the publication database table.
Step 9: After the row content of all publication tables is consistent across the publication and subscription databases, start the streaming.
The -pubname option of the startstreaming command specifies the publication name.

The two databases being compared are referred to as the source database and the target database. The source database can be of type EnterpriseDB. The target database must also be type EnterpriseDB.
Note: The Data Validator does not validate columns having the following data types. Tables containing one or more columns of these types will only be partially validated.
•
•
•
•
•
•
•
•
Note: Make sure that the data streaming between the source and target EDB Postgres Replication Server tables has been completed before using the Data Validator in EDB Postgres Replication Server (single-master or multi-master). If streaming is still in progress, it is possible that the Data Validator will show differences in tables.
Step 1: When you install the EDB Postgres Replication Server, the components for the Data Validator are installed as well. See Chapter 3 for information on installing the EDB Postgres Replication Server.
Note: EPRS_HOME is the directory where EDB Postgres Replication Server is installed. This may or may not be the same as the Postgres home directory depending upon how EDB Postgres Replication Server is installed.
Step 3: Edit the datavalidator.properties file located in the EPRS_HOME/ datavalidator/etc directory and specify the connection information for the source and target databases you want to compare.
The following are the parameters in the datavalidator.properties file.
The following is the initial content of the datavalidator.properties file after installation:
Step 4: Determine the location for the Data Validator logs directory.
The Data Validator generates a log file with a name formatted as datavalidator_yymmdd-hhmiss.log in the logs directory for each run.
If there are row differences between the source and target tables, a file with a name formatted as datavalidator_yymmdd-hhmiss.diff is also generated that contains output of the errors in diff format. Use a graphical diff tool like Kompare to view this file to highlight the specific differences.
Data Validator attempts to create a subdirectory named logs within the EPRS_HOME/ datavalidator/bin directory the first time you invoke the Data Validator without the -ld option. If you do not invoke the Data Validator as the root account, it is likely that the run will fail as it attempts to create subdirectory logs in the EPRS_HOME/ datavalidator/bin directory where typically only the root account has this privilege.
•
Run the Data Validator as the root account. This enables the Data Validator to create the logs subdirectory within the EPRS_HOME/datavalidator/bin directory, and then to create the log and diff files in the logs subdirectory.
•
Create the EPRS_HOME/datavalidator/bin/logs directory structure before running the Data Validator. Modify the permissions on directory EPRS_HOME/datavalidator/bin/logs so the operating system account you use to run the Data Validator has the privilege to create files in the directory.
•
Use the -ld log_directory_path option to allow the Data Validator to create the log and diff files in the specified directory location log_directory_path. Be sure the operating system account you use to run the Data Validator has the proper privileges to either create the lowest level subdirectory specified by log_directory_path if it does not already exist or to create files within the specified directory if the full directory path already does exist.
The current working directory from which you invoke the Data Validator script runValidation.sh must be the bin subdirectory containing the script (that is, EPRS_HOME/datavalidator/bin).
[ option ] ...
schema_name is the name of the schema in the source database containing the tables to be validated. The choices for option are listed later in this section within the Options subsection.
The general syntax for --version and --help is shown by the following:
[ -ts schema ]
[ -it table_1 [,table_2 ] ... ]
[ -et table_1 [,table_2 ] ... ]
[ -ld log_directory_path ]
[ -ds { true | false } ]
[ -sdbms database_type ]
[ -sh host ]
[ -sp port ]
[ -sdb dbname ]
[ -su user ]
[ -spw password ]
[ -tdbms database_type ]
[ -th host ]
[ -tp port ]
[ -tdb dbname ]
[ -tu user ]
[ -tpw password ]
[ -bs row_count ]
[ -fs row_count ]
For clarity, the preceding syntax diagram shows only the single-character form of the option. The Options subsection lists both the single-character and multi-character forms of the options.
Specification of any database connection option (-sdbms through -tpw listed in the preceding syntax diagram) overrides the corresponding parameter in the datavalidator.properties file. See Section 7.1 for information on the datavalidator.properties file.
-ss, --source-schema schema
-ts, --target-schema schema
-it, --include-tables table_1 [,table_2 ] ...
-et, --exclude-tables table_1 [,table_2 ] ...
The tables within the source schema that are to be excluded from comparison. If omitted, only those tables specified with the -it option are included for comparison. If both the -it and -et options are omitted, all source schema tables are included for comparison. Note: There must be no white space between the comma and table names.
-srs, --skip-rowsonlyin-source { true | false }
When true is specified, the logging of differences for rows that exist only in the source database table are skipped. The default is false.
-srt, --skip-rowsonlyin-target { true | false }
When true is specified, the logging of differences for rows that exist only in the target database table are skipped. The default is false.
-srb, --skip-rowsin-both { true | false }
When true is specified, the logging of differences for rows that exist both in the source and target database tables with the same primary key, but with different non-primary key values are skipped. The default is false.
-ld, --logging-dir log_directory_path
Directory path to where the Data Validator log and diff files are to be created and stored. If log_directory_path does not exist, Data Validator attempts to create it. If a full directory path is not specified log_directory_path is created or assumed to be located relative to the EPRS_HOME/datavalidator/ bin subdirectory where the runValidation.sh script is invoked. (That is, the logs directory is EPRS_HOME/datavalidator/bin/log_directory_path.) Be sure the operating system account used to invoke the runValidation.sh script has the privileges to create the directory if it does not already exist, or to create files in the specified directory if it does already exist. If omitted, the default is the EPRS_HOME/bin/logs directory.
-ds, --display-summary { true | false }
Specify true to display only the Data Validator summary. This omits the source and target database connection information as well as the detailed breakdown of the results by source database table. Specify false to display all of the Data Validator results. The type and amount of information that is displayed at the command line console when the Data Validator is invoked is the same information that is also stored in the log file for that run. If omitted, the default is false (that is, all of the Data Validator results is displayed).
-sdbms, --source-dbms database_type
The type of the source database server. Supported types are oracle, enterprisedb, sqlserver, sybase, and mysql.
-sh, --source-host host
-sp, --source-port port
The port number on which the source database server is listening for connections.
-sdb, --source-database dbname
-su, --source-user user
-spw, --source-password password
-tdbms, --target-dbms database_type
-th, --target-host host
-tp, --target-port port
The port number on which the target database server is listening for connections.
-tdb, --target-database dbname
-tu, --target-user user
-tpw, --target-password password
-bs, --batch-size row_count
The -bs option specifies the number of rows to group in a batch to be used for comparison across the source and target database tables. For example, if a table contains 1000 rows, then a -bs setting of 100 requires 10 batch iterations to complete the comparison across the source and target databases. The Data Validator reads 100 rows, both from the source and target tables, and adds them in source and target buffers. The validation thread then reads the 100 rows from the source and target buffers and performs the comparison. It will then move to read and prepare the next 100 rows for comparison and so on. Note that the actual database round trips required to bring in 100 rows from the database depends on the -fs option for the fetch size. For example, an -fs setting of 100 needs just one round trip whereas an -fs setting of 10 requires 10 database round trips.
-fs, --fetch-size row_count
Performing data validation for tables that are quite large in size may cause the Data Validator to terminate with an out of heap space error when using the default fetch size of 5000 rows. Use the -fs option to specify a smaller fetch size to help avoid the out of heap space issue. The result set iteration will bring in as many rows as represented by the row_count value in a single database round trip.
The following lists the tables in schema EDB along with the content of tables DEPT and EMP in the source database:
The following lists the tables in the schema public along with the content of tables dept and emp in the Advanced Server edb target database:
•
The Source DEPT table contains one extra row with DEPTNO 50 that does not exist in the target Advanced Server dept table.
•
The rows in the EMP table with EMPNO values 9001 and 9002 have column values that differ between the Source and target Advanced Server tables
•
In this example, the JOBHIST table contains identical rows for both the source and target Advanced Server tables.
The content of the datavalidator.properties file is set as follows:
The following example compares all tables in the EDB schema against the public schema.
The Data Validator log files are created in directory /EPRS_home/datavalidator/bin/logs/ as specified with the -ld option. The operating system account used to invoke the runValidation.sh script has write access to the EPRS_home directory so the Data Validator can create the /EPRS_home/datavalidator/bin/logs subdirectory.
All tables count: 3
Validated tables count: 3
Rows count: 38
Errors count: 3
Tables having only unsupported datatypes count: 0
Tables having primary key limitation count: 0
Total time(s): 0.678
Rows per second: 56
•
There is one error in the DEPT table (the missing row).
•
There are two errors in the EMP table (the two rows with mismatching column values)
•
The JOBHIST table contains no errors.

A control replication origin will be auto-created as part of the publication database registration process. The origin will be named after database name that is _ngx_DBNAME for example _ngx_inventory. The control replication origin will be removed once the publication database is unregistered.
Note: In case the replication origin fails to create or remove, there will not be any impact on the relevant adddb or removedb operation and only a warning message will be logged.
Step 1: Open a SQL terminal (for example psql) and start the user transaction.
Step 2: Setup replication origin session for the current transaction, before performing any other (query) operation.
Step 3: Make changes in the application by running user-specific queries (intended for bulk change).
Add multiple rows in the table exclude_user_test which is also a part of the Publication. Simulate bulk changes (5K) that are to be skipped from replication. This table will also be part of EPRS7 cluster Publication.
Step 4: Reset replication origin.

1.
Error: Set admin password first before execution of other command(s).
Cause: The admin password is not set before adding a database to the replication server.
Workaround: Set the admin password before adding a database to the replication server.
2.
Error: Unable to encrypt password. Reason Invalid user name or password.
Cause: Incorrect username or password given.
Workaround: Provide the correct username or password.
3.
Error: Error encountered: Failed to add database with id database_Id.
Causes: This error can occur in the following scenarios
Workaround: Provide the correct encrypted password. Provide the correct IP address (IP address of the database host). Check if the database server is running or not. Encrypt the database password using encrypt command and use the same while adding the database.
4.
Error: Failed to add database with id db1. WARNING: Connection error: FATAL: no pg_hba.conf entry for host "x.x.x.x", user "xxx", database "database_Id", SSL off.
Cause: The IP address for the database host is not present in pg_hba.conf file.
Workaround: Add the IP address of the database host in the pg_hba.conf file located at var/lib/edb/asx/. For a cluster the pg_hba.conf file is located at /var/lib/edb/asx/clusterx.
asx is the EDB Advanced Version and
clusterx is the cluster on which the database is running, for example, cluster1 or cluster2 and so on.
5.
Error: The database id database_Id is not registered with the network.
Cause: The database does not exist or an incorrect database id is provided.
Workaround: Provide the correct database id or create the database with the create database command.
6.
Error: The server server_name is not registered with the network.
Cause: Incorrect servername is provided or the server does not exist in the network.
Workaround: Provide the correct servername or register the server in the network with the joinnetwork command.
7.
Error: Publication publication_name not found.
Snapshot failed for Publication publication_name to target database database_Id.
Cause: Publication does not exist or an incorrect publication name is given.
Workaround: Provide the correct publication name or create a publication.
8.
Error: Subscription with DB id database_Id not found for Publication publication_name.
Snapshot failed for Publication publication_name to target database database_Id.
Cause: Incorrect database_Id provided or the database does not exist.
Workaround: Provide the correct database_Id or create the database if it does not exist.
9.
Error: Snapshot cannot be performed, Publication database id database_Id is same as the Consumer database id database_Id.
Cause: The publication database is the same as the consumer database.
Workaround: The publication and consumer database should be different.
10.
Error: One or more Subscriptions are associated with Publication publication_name. Please un-subscribe (via leavepub command) before removing the Publication.
Cause: removepub command is run before the publication leaves the network.
Workaround: Run leavepub command for all the subscription databases before removing the publication.
Error: Cannot register database because it is already registered by a publication service.
Cause: Database can be registered with the replication cluster only once.
Workaround: Database is already registered. You can create a publication with required tables with the createpub command.
Error: Exception in thread "main" javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused
Cause: Occurs whenever a connection cannot be made to the EDB Replication Server.
Workaround: Check that you have entered the correct host IP address and port number of the server. Check that the server is running. Check that in the pg_hba.conf file, the hostname is mapped to the correct network IP address, which matches the IP address returned by the Linux ifconfig command. Check if the publication server has access to the database in the pg_hba.conf file.
Error: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Cause: Occurs when attempting to save a publication database definition. The publication server cannot connect to the database server network location.
Workaround: Verify that the correct IP address and port for the database server are given. Verify that the database server is running and is accessible from the host running the publication server.
Error: Could not connect to the database server. Reason: FATAL: number of requested standby connections exceeds max_wal_senders (currently n)
Cause: Occurs when attempting a snapshot replication from a publication database which is by default configured with the log-based method of synchronization replication (that is, WAL based logical replication), and the additional concurrent connection for logical replication exceeds the current setting, n, of the max_wal_senders configuration parameter in the postgresql.conf file.
Workaround: Increase the value of max_wal_senders in the postgresql.conf file for the database server running the publication database. Restart the database server containing the publication database.
The path for the postgresql.conf file is /var/lib/edb/asx/clusterx.
asx is the EDB Advanced Version and
clusterx is the cluster on which the database is running, for example, cluster1 or cluster2 and so on.
Error: Currently no publication exists on the publication server. Please create at least one publication on the server and then retry.
Cause: If you join a publication when there are no publications on the specified publication server, then this error message is thrown.
Workaround: Create a publication with the createpub command and then join a publication.
Error: Database cannot be removed. Reason: Publication database connection cannot be removed as one or more publications are defined against it.
Cause: There are existing publications on the database.
Workaround: Make sure all the publications pertaining to the publication database have been removed. Use leavepub command to leave the publication and removepub command to remove the publication.
Error: Database connection cannot be added. FATAL: no pg_hba.conf entry for host "xxx.xxx.xx.xxx", user "user_name", database "database_name", SSL off
Cause: Occurs when attempting to save a database definition using the adddb command.
Workaround: Verify that the database host IP address, port number, database user name, password, and database identifier are correct. Verify there is an entry in the pg_hba.conf file permitting access to the database by the given user name originating from the IP address where the EDB Replication Server is running.
Error: Filter cannot be defined for Binary data type column(s) e.g. BYTEA, BLOB, RAW.
Cause: Occurs when attempting to define a filter rule on a column with a binary data type in a publication table. Filter rules are not permitted on such columns.
Workaround: Do not add filters on binary data type columns.
Error: Filter with same name/clause already exist on table/view: schema.table_name
Cause: When adding a filter rule on a publication table, the same filter name or the same filter clause (WHERE clause) cannot be used more than once on a given table.
Workaround: Modify the duplicate filter name or filter clause so it is unique for the table.
Error: The triggers creation failed for one or more publication tables. Make sure the database is in valid state and user is granted the required privileges.
Cause: Either the user does not have the trigger creation privilege or there is a database server problem. The database server message is displayed as part of the error.
Workaround: Provide a username with sufficient privileges while adding a database.
Error: Problem occurred in publish process. Reason: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Resolution: Occurs when attempting synchronization replication and the controller database is not accessible by the publication server.
Workaround: Verify that the correct IP address and the port have been defined in the publication database definition of the controller database. Verify that the database server is running and is accessible from the host running the publication server.
You can get an error that the Publication cannot be created for the following scenarios:
1.
Error: Publication cannot be created. Publication publication_name already exists on the publisher server. Please choose a different name and then proceed.
Cause: Publication names must be unique within a publication server.
Workaround: Enter a different publication name.
2.
Error: Publication cannot be created. Table schema.table_name replica identity is set to replica_identity_setting. To define a Filter, the table replica identity should be set to FULL.
Cause: Occurs when a table filter is attempted to be defined on a publication table used in a log-based replication system.
Workaround: Use the ALTER TABLE statement to change REPLICA IDENTITY to FULL.
3.
Error: Publication cannot be created. Table table_name does not contain a primary key. Transactional replication is not supported for a non-pk table.
Cause: All tables used for synchronization replication must have primary keys.
Workaround: Create a primary key on the table.
Error: The publication schema cannot be created. Reason: ERROR: Permission denied for database db_name.
Cause: Occurs when attempting to create the publication database definition and the specified publication database user does not have the privilege to create a schema in database db_name.
Workaround: Grant the CREATE privilege on the database to the publication database user.
Error: A replication slot is not available on the target database server. Please configure the max_replication_slots GUC on the database server.
Cause: Occurs when attempting to add a publication database definition with the log-based method of synchronization replication, and the max_replication_slots configuration parameter in the postgresql.conf file is not set to a large enough value to accommodate the additional database.
Workaround: Increase the value of the max_replication_slots parameter and restart the database server.
Error: main" javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
Cause: EDB Replication Server is not running.
pkill -f 'ngen'
5.
Verify the following values in /usr/edb/rs-7.0/server/etc/application.properties file:
Note: Make sure that the following directories are present after installing EDB Replication Server.
/var/lib/edb/asx/data/log
Step 1: Verify that the database servers participating in the replication cluster are all running.
Step 2: Verify that the EDB Replication Server is running.
Step 3: For the master definition node in a multi-master replication system, verify that the publication database user is a superuser and has the privilege to modify pg_catalog tables.
Step 4: Verify that the network IP address returned by the ifconfig command matches the IP address set for ngen.server.host in application.properties.
Step 5: Verify that the ports are not preoccupied. If the ports are preoccupied change the ports in application.properties and server.properties. Refer to section 2.1.5 for more details.
Step 6: Verify that sufficient free disk space is available otherwise EDB Replication Server will run into problems while installing and configuring.
Step 7: If you have configured a firewall make sure that the firewall settings allow access to the ports used by EDB Replication Server in a cluster setup with EDB Replication Servers running on different machines.
Step 8: For a geographically distributed network make sure to substantially increase the value of the parameter zookeeper.session.timeouts in the server.properties file located at /usr/edb/rs-x.0/server/etc. Otherwise, there would be frequent outages between the broker and the zookeeper. The default value is 6000.
Step 1: Check the ngx-server.log file for errors.
Step 2: Check the log file of the database server running the controller database for errors.

•
Step 1: Add a new partition to an existing publication partition table.
The following is the syntax is for range partitioning with sub partitioning as list:
Example: The following is an example of an MMR with three nodes cluster node1, node2 and, node3. In this case partition by range is used with sub partition as list.
4.
Create publication on one of the nodes and then use joinpub command for the other nodes.
6.
Check that the events_queue table is present across all the nodes:
Step 2: Insert or update the data in the partitioned table. Check if the changes in the partition table replicate to the target database table(s) as well.
Node2:
Step 3: Verify that after adding the database the following control objects are created under _ngx_rep_cluster control schema in the given publication database.
Step 4: Verify that after removing the database the following control objects are removed from _ngx_rep_cluster control schema in the given publication database.