Chat now with support
Chat with Support

SharePlex 12.0 - Installation and Setup guide

About this Guide Conventions used in this guide Installing and Setting up SharePlex on Oracle Source Database
SharePlex Pre-installation Checklist for Oracle Download the SharePlex installer Install SharePlex on Linux and UNIX Set up an Oracle environment for replication Set up replication from Oracle to a different target type Installation and Setup for Cloud-Hosted Databases for Oracle Installation and setup for remote capture Installation and setup for HA cluster Generic SharePlex demonstration for Oracle Advanced SharePlex demonstrations for Oracle Database Setup Utilities Solve Installation Problems for Oracle
Installing and Setting up SharePlex on a PostgreSQL Database as Source and Service
SharePlex Pre-installation Checklist for PostgreSQL Download the SharePlex installer for PostgreSQL Install SharePlex on Linux for PostgreSQL as a Source Set up Replication from PostgreSQL to Supported Target Types Installation and Setup for Cloud-Hosted Databases for PostgreSQL Installation and Setup for Remote Capture for PostgreSQL Install SharePlex on PostgreSQL High Availability Cluster Configure SharePlex on PostgreSQL Azure Flexible Server with High Availability Using Logical Replication Generic SharePlex Demonstration for PostgreSQL Advanced SharePlex Demonstrations for PostgreSQL Database Setup for PostgreSQL Database Setup for PGDB as a Service Installation of pg_hint_plan extension Solve Installation Problems for PostgreSQL
Installing SharePlex on a Docker container Assign SharePlex users to security groups Solve Installation Problems Uninstall SharePlex Advanced installer options Install SharePlex as root SharePlex installed items

Set up Replication from PostgreSQL to Supported Target Types

Contents
About these instructions

This chapter contains instructions for configuring SharePlex to replicate from PostgreSQL to a different type of target. This is known as heterogeneous replication.

These instructions highlight specific tasks that are pertinent to the flow of data between source and target. Refer to other topics in the SharePlex documentation as needed to complete the configuration, deploy any optional features that apply, and monitor and maintain the environment.

For additional information, see:

  • For the SharePlex-supported datastores, data types and operations that are supported by SharePlex, see the "System Requirements " section of SharePlex Release Notes.
  • For additional configuration options, activation steps, and monitoring information, see SharePlex Administration Guide.
  • For reference documentation on SharePlex commands, parameters and utilities, see SharePlex Reference Guide.

Set up replication from PostgreSQL to PostgreSQL

Note: These instructions apply to all implementations of PostgreSQL unless otherwise noted.

Overview

SharePlex can post replicated PostgreSQL data to a PostgreSQL target database through an Open Database Connectivity (ODBC) interface. SharePlex supports all implementations of the PostgreSQL open-source database.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, data types and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Configure SharePlex on the source and target

Set up SharePlex and the database on the PostgreSQL source and target system as follows.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information.

Datasource:r.source_DB

src_schema.table

tgt_schema.table

host@r.database_name

where:

  • source_DB is the database name of the source database.
  • src_schema.table is the schema and name of the source table.
  • tgt_schema.table is the schema and name of the target table.*
  • host is the name of the target system.
  • database_name is the name of the target database.
* Important! 

Target schema or table name defined in the database is case sensitive, be certain to:

  • Type the name in the correct case.
  • Enclose the name in quotation marks, for example "MySchema"."MyTable".

Note: This is a basic one-source, one-target configuration using no additional SharePlex configuration features. See "Configure SharePlex to replicate data" in the SharePlex Administration Guide for important information about creating a configuration file and for additional setup instructions for more complex replication scenarios.

Source configuration example

The following configuration file replicates table HR.Emp from PostgreSQL instance dbname112 to target table region1.emp in database mydb on target system hostB. The source table is case-sensitive.

Datasource:r.dbname112

HR."Emp" region1.emp hostB@r.mydb

Set up replication from PostgreSQL to Oracle

Overview

SharePlex can post replicated PostgreSQL data to an Oracle target database through an Open Database Connectivity (ODBC) interface. SharePlex supports all implementations of the PostgreSQL open-source database.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, data types and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Configure SharePlex on the source and target

Set up SharePlex and the database on the PostgreSQL source and an Oracle target system as follows.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information.

Datasource:r.dbname

src_schema.table

tgt_owner.table

host@o.database_name

where:

  • source_DB is the database name of the source database.
  • src_schema.table is the schema and name of the source table.
  • tgt_owner.table is the schema and name of the target table.*
  • host is the name of the target system.
  • database_name is the name of the target database.
* Important! 

Target schema or table name defined in the database is case sensitive, be certain to:

  • Type the name in the correct case.
  • Enclose the name in quotation marks, for example "MySchema"."MyTable".

Note: This is a basic one-source, one-target configuration using no additional SharePlex configuration features. See "Configure SharePlex to replicate data" in the SharePlex Administration Guide for important information about creating a configuration file and for additional setup instructions for more complex replication scenarios.

An example of a config file for PostgreSQL to Oracle replication:

# data2k configuration file

datasource:r.source_databasename

"demo"."data2k" "DEMO"."DATA2K" target_system@o.target_databasename

Set up replication from PostgreSQL to Kafka

Overview

The SharePlex Post process can connect and write to a Kafka broker. The data can be written in JSON or XML output as a sequential series of operations as they occurred on the source, which can then be consumed by a Kafka consumer.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, data types and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Kafka category

Supported targets

Kafka

Command options

Use the below command options on the SP_CTRL for Kafka:

target x.kafka set kafka broker= 10.250.40.42:9092 (IP address of Kafka server:port number)

target x.kafka set kafka topic= kafkaTarget (topic name)

target x.kafka set format record = xml/JSON

target x.kafka show

Guidelines for posting to Kafka

  • A SharePlex Post process acts as a Kafka producer. A SharePlex Post process can write to one or more topics that have one or more partitions.
  • The SharePlex Post process does not create a topic itself, but you can configure the Kafka broker to auto-create topics.

Configure SharePlex on the source

When replicating data to Kafka, configure the source database and SharePlex on the source system as follows.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information. The structure that is required in a configuration file varies, depending on your replication strategy, but this shows you the required syntax for routing data to Kafka.

Datasource:r.dbname
src_schema.table !kafka host

where:

  • dbname is a PostgreSQL database name of source
  • src_schema.table is the schema and name of the source table.
  • !kafka is a required keyword indicating SharePlex is posting to Kafka.

  • host is the name of the target system.

Note: See Configure SharePlex to replicate data in the SharePlex Administration Guide.

Source configuration example

Datasource:r.testdb

MY_SCHEMA.MY_TABLE !kafka targetHost

Configure SharePlex on the target

These instructions configure the SharePlex Post process to connect to Kafka. You must have a running Kafka broker.

Perform these steps to configure post to Kafka:

  1. Create a Kafka topic.
  2. Start sp_cop. (Do not activate the configuration yet.)
  3. Run sp_ctrl.

  4. Issue the target command to configure posting to a Kafka broker and topic. The following are example commands.

    sp_ctrl> target x.kafka set kafka broker=10.250.40.42:9092

    sp_ctrl> target x.kafkaset kafka topic=shareplex

    See Set up replication from PostgreSQL to Kafka for command explanations and options.

Set the Kafka Record Format

SharePlex can output to either XML or JSON format as input to Kafka. XML is the default. To set the input format and specify format options, use one of the following target commands:

target x.kafka set format record=json

or:

target x.kafka set format record=xml

To view samples of these formats, see the format category of the target command documentation in the SharePlex Reference Guide.

View and change Kafka settings

To view current property settings for output to Kafka, use the following target command:

target x.kafka show

Table 13: Kafka target properties

Property Input Value Default
broker=broker

Required. The host and port number of the Kafka broker, or a comma delimited list of multiple brokers. This list is the bootstrap into the Kafka cluster. So long as Post can connect to one of these brokers, it will discover any other brokers in the cluster.

 localhost:9092
client_id=ID

Optional. A user-defined string that Post will send in each request to help trace calls.

None
compression.code={none, gzip, snappy} Optional. Controls whether data is compressed in Kafka. Options are none, gzip or snappy. None
partition={number | rotate | rotate trans}

Required. One of the following:

  • A fixed partition number: Directs Post to post messages only to the specified partition number. For example, setting it to 0 directs Post to post only to partition 0. This option is suitable for use in testing or if the target has multiple channels of data posting to the same Kafka topic.
  • The keyword rotate: Directs Post to apply messages to all of the partitions of a topic in a round-robin fashion. The partition changes with each new message. For example if a topic has three partitions, the messages are posted to partitions 0,1,2,0,1,2, and so on in that order.
  • The keyword rotate trans: This is similar to the rotate option, except that the partition is incremented with each transaction rather than with each message. For example, if a topic has three partitions, the messages are posted to partition 0 until the commit, then to partition 1 until the commit, and so on in that order. This option is suitable if you are replicating multiple tables to a single topic. It allows you to distribute data across several partitions, while still preserving all of the operations of a transaction together in a single partition. This enables a consumer that reads from a single partition to receive a stream of complete transactions.
0
request.required.acks=value

Optional. This is a Kafka client parameter. By default it is set to a value of -1, which means all. Consult the Kafka documentation about this subject, because all really means all in-sync replicas. This parameter can be used in conjunction with the min.insync.replicas broker parameter to tune behavior between availability and data consistency.

Important: It is possible for data to be lost between a Kafka producer (SharePlex in this case) and a Kafka cluster, depending on these settings.

 -1
topic=topic_name

Required. The name of the target Kafka topic.

This string may contain the special sequences %o or %t. The %o sequence is replaced by the schema name of the table that is being replicated. The %t sequence is replaced by the table name of the table that is being replicated. This feature may be used in conjunction with a Kafka server setting of auto.create.topics.enable set to 'true'. Also view your server settings for default.replication.factor and num.partitions because these are used as defaults when topics are auto created.

Important! If using multiple topics, you must also set the following properties with the target command:

  • The output must be in JSON. Set the record property of the format category to json:

    target x.kafka set format record=json

  • Commits must be disabled. Set the commit property of the json category to no:

    target x.kafka set json commit=no

 shareplex

* To avoid latency, if Post detects no more incoming messages, it sends the packet to Kafka immediately without waiting for the threshold to be satisfied.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating