Chat now with support
Chat with Support

SharePlex 11.4 - Installation and Setup guide

About this Guide Conventions used in this guide Installing and Setting up SharePlex on an Oracle Source
SharePlex Pre-installation Checklist for Oracle Download the SharePlex installer Install SharePlex on Linux and UNIX Set up an Oracle environment for replication Set up replication from Oracle to a different target type Installation and Setup for Cloud-Hosted Databases for Oracle Installation and setup for remote capture Installation and setup for HA cluster Generic SharePlex demonstration for Oracle Advanced SharePlex demonstrations for Oracle Database Setup Utilities Solve Installation Problems for Oracle
Installing and Setting up SharePlex on a PostgreSQL Database as Source and Service
SharePlex Pre-installation Checklist for PostgreSQL Download the SharePlex installer for PostgreSQL Install SharePlex on Linux for PostgreSQL as a Source Set up Replication from PostgreSQL to Supported Target Types Installation and Setup for Cloud-Hosted Databases for PostgreSQL Installation and Setup for Remote Capture for PostgreSQL Install SharePlex on PostgreSQL High Availability Cluster Configure SharePlex on PostgreSQL Azure Flexible Server with High Availability Using Logical Replication Generic SharePlex Demonstration for PostgreSQL Advanced SharePlex Demonstrations for PostgreSQL Database Setup for PostgreSQL Database Setup for PGDB as a Service Installation of pg_hint_plan extension Solve Installation Problems for PostgreSQL
Installing SharePlex on a Docker container Assign SharePlex users to security groups Solve Installation Problems Uninstall SharePlex Advanced installer options Install SharePlex as root SharePlex installed items

Set up replication from PostgreSQL to Kafka

Overview

The SharePlex Post process can connect and write to a Kafka broker. The data can be written in JSON or XML output as a sequential series of operations as they occurred on the source, which can then be consumed by a Kafka consumer.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, data types and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Kafka category

Supported targets

Kafka

Command options

Use the below command options on the SP_CTRL for Kafka:

target x.kafka set kafka broker= 10.250.40.42:9092 (IP address of Kafka server:port number)

target x.kafka set kafka topic= kafkaTarget (topic name)

target x.kafka set format record = xml/JSON

target x.kafka show

Guidelines for posting to Kafka

  • A SharePlex Post process acts as a Kafka producer. A SharePlex Post process can write to one or more topics that have one or more partitions.
  • The SharePlex Post process does not create a topic itself, but you can configure the Kafka broker to auto-create topics.

Configure SharePlex on the source

When replicating data to Kafka, configure the source database and SharePlex on the source system as follows.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information. The structure that is required in a configuration file varies, depending on your replication strategy, but this shows you the required syntax for routing data to Kafka.

Datasource:r.dbname
src_schema.table !kafka host

where:

  • dbname is a PostgreSQL database name of source
  • src_schema.table is the schema and name of the source table.
  • !kafka is a required keyword indicating SharePlex is posting to Kafka.

  • host is the name of the target system.

Note: See Configure SharePlex to replicate data in the SharePlex Administration Guide.

Source configuration example

Datasource:r.testdb

MY_SCHEMA.MY_TABLE !kafka targetHost

Configure SharePlex on the target

These instructions configure the SharePlex Post process to connect to Kafka. You must have a running Kafka broker.

Perform these steps to configure post to Kafka:

  1. Create a Kafka topic.
  2. Start sp_cop. (Do not activate the configuration yet.)
  3. Run sp_ctrl.

  4. Issue the target command to configure posting to a Kafka broker and topic. The following are example commands.

    sp_ctrl> target x.kafka set kafka broker=10.250.40.42:9092

    sp_ctrl> target x.kafkaset kafka topic=shareplex

    See Set up replication from PostgreSQL to Kafka for command explanations and options.

Set the Kafka Record Format

SharePlex can output to either XML or JSON format as input to Kafka. XML is the default. To set the input format and specify format options, use one of the following target commands:

target x.kafka set format record=json

or:

target x.kafka set format record=xml

To view samples of these formats, see the format category of the target command documentation in the SharePlex Reference Guide.

View and change Kafka settings

To view current property settings for output to Kafka, use the following target command:

target x.kafka show

Table 13: Kafka target properties

Property Input Value Default
broker=broker

Required. The host and port number of the Kafka broker, or a comma delimited list of multiple brokers. This list is the bootstrap into the Kafka cluster. So long as Post can connect to one of these brokers, it will discover any other brokers in the cluster.

 localhost:9092
client_id=ID

Optional. A user-defined string that Post will send in each request to help trace calls.

None
compression.code={none, gzip, snappy} Optional. Controls whether data is compressed in Kafka. Options are none, gzip or snappy. None
partition={number | rotate | rotate trans}

Required. One of the following:

  • A fixed partition number: Directs Post to post messages only to the specified partition number. For example, setting it to 0 directs Post to post only to partition 0. This option is suitable for use in testing or if the target has multiple channels of data posting to the same Kafka topic.
  • The keyword rotate: Directs Post to apply messages to all of the partitions of a topic in a round-robin fashion. The partition changes with each new message. For example if a topic has three partitions, the messages are posted to partitions 0,1,2,0,1,2, and so on in that order.
  • The keyword rotate trans: This is similar to the rotate option, except that the partition is incremented with each transaction rather than with each message. For example, if a topic has three partitions, the messages are posted to partition 0 until the commit, then to partition 1 until the commit, and so on in that order. This option is suitable if you are replicating multiple tables to a single topic. It allows you to distribute data across several partitions, while still preserving all of the operations of a transaction together in a single partition. This enables a consumer that reads from a single partition to receive a stream of complete transactions.
0
request.required.acks=value

Optional. This is a Kafka client parameter. By default it is set to a value of -1, which means all. Consult the Kafka documentation about this subject, because all really means all in-sync replicas. This parameter can be used in conjunction with the min.insync.replicas broker parameter to tune behavior between availability and data consistency.

Important: It is possible for data to be lost between a Kafka producer (SharePlex in this case) and a Kafka cluster, depending on these settings.

 -1
topic=topic_name

Required. The name of the target Kafka topic.

This string may contain the special sequences %o or %t. The %o sequence is replaced by the schema name of the table that is being replicated. The %t sequence is replaced by the table name of the table that is being replicated. This feature may be used in conjunction with a Kafka server setting of auto.create.topics.enable set to 'true'. Also view your server settings for default.replication.factor and num.partitions because these are used as defaults when topics are auto created.

Important! If using multiple topics, you must also set the following properties with the target command:

  • The output must be in JSON. Set the record property of the format category to json:

    target x.kafka set format record=json

  • Commits must be disabled. Set the commit property of the json category to no:

    target x.kafka set json commit=no

 shareplex

* To avoid latency, if Post detects no more incoming messages, it sends the packet to Kafka immediately without waiting for the threshold to be satisfied.

Set up Replication from PostgreSQL to SQL Server

Overview

SharePlex can post replicated source data to a Microsoft SQL Server target database through an Open Database Connectivity (ODBC) interface.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, data types and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Configure SharePlex on the source

When replicating data to SQL Server, configure the source database and SharePlex on the source system as follows.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information. The structure that is required in a configuration file varies, depending on your replication strategy, but this shows you the required syntax for routing data to SQL Server.

Datasource:r.dbname
src_schema.table tgt_owner.table host

where:

  • dbname is a PostgreSQL database name of source
  • src_schema.table is the schema and name of the source table.

  • tgt_owner.table is the owner and name of the target table.

  • host is the name of the target system.

Note: See Configure SharePlex to replicate data in the SharePlex Admin Guide.

Source configuration example

The following example replicates table SCOTT.EMP from PostgreSQL instance dbnameA to the MS SQL Server on host sysprod

Datasource: r.testdb
SCOTT.EMP    SCOTT.EMP   sysprod@r.mydb

Configure SharePlex on the target

Perform the following steps to configure SharePlex on the target:

  1. Make certain that the database setup meets all of the requirements in Open target checklist .

  2. Run Database Setup for SQL Server (mss_setup) to establish a database account and connection information for SharePlex. For more information, see Database setup for SQL Server.

Installation and Setup for Cloud-Hosted Databases for PostgreSQL

Contents

 

Overview of SharePlex Setup on Cloud for PostgreSQL

SharePlex supports databases installed as services of Amazon Web Services (AWS), Google Cloud SQL for PostgreSQL, and Microsoft Azure. To view the cloud databases that SharePlex supports, refer to the Supported Cloud Platforms section in the SharePlex Release Notes document for the respective databases.

There are some differences in the way that SharePlex installs in an IaaS cloud environment, a PaaS cloud environment, and a SaaS cloud environment. These differences are only in the installation and configuration of SharePlex. Once installed and configured, SharePlex operates in the cloud the same way that it operates in on-premise installations. For detailed information regarding how SharePlex is installed in various environments, see Overview of SharePlex Setup on Cloud.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating