Chat now with support
Chat with Support

SharePlex 9.0.2 - Installation Guide

About this Guide Conventions used in this guide System Requirements SharePlex pre-installation checklist Installation and setup for Oracle cluster Installation and setup for cloud-hosted databases Download SharePlex Install SharePlex on Linux and UNIX Install SharePlex on Windows Assign SharePlex users to security groups Set up an Oracle environment for replication Set up replication between SQL Server databases Set up replication between different database types Basic SharePlex demonstrations Advanced SharePlex demonstrations Solve Installation Problems Uninstall SharePlex SharePlex Utilities Advanced installer options Install SharePlex as root Run the installer in unattended mode SharePlex installed items

Set up replication from Oracle to Teradata

Overview

SharePlex can post replicated Oracle data to a Teradata target database through an Open Database Connectivity (ODBC) interface.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, datatypes and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Review column names

To support replication between a source of one database type and a target of another type, the letter case of the names of the source and target columns must be the same, for example the column names on both sides in lower case or both sides in upper case. If the case differs between the source and target column names, use the column mapping feature to map the column names in the configuration file.

See the SharePlex Administration Guide for more information about column mapping with SharePlex.

Configure SharePlex on the source

Set up SharePlex and the database on the Oracle source system as follows.

Make keys available to SharePlex

To replicate from an Oracle source to an Open Target target, you must make key information available to SharePlex.

Enable Oracle supplemental logging

Enable PK/UK supplemental logging in the Oracle source database. SharePlex must have the Oracle key information to build an appropriate key on the target.

Set SP_OCT_USE_SUPP_KEYS parameter

Set the SP_OCT_USE_SUPP_KEYS parameter to a value of 1. This parameter directs SharePlex to use the columns set by Oracle's supplemental logging as the key columns when a row is updated or deleted. When both supplemental logging and this parameter are set, it ensures that SharePlex can always build a key and that the SharePlex key will match the Oracle key.

See the SharePlex Reference Guide for more information about this parameter.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information.

Datasource:o.SID

src_owner.table

tgt_owner.table

host@r.database_name

where:

  • SID is the Oracle SID of the source Oracle database..
  • src_owner.table is the owner and name of the source table.
  • tgt_owner.table is the owner and name of the target table.*
  • host is the name of the target system.
  • r. identifies the target as non-Oracle.
  • database_name is the name of the target database. Important! database_name must be the actual name of the database, not a data source name (DSN).

* Important! 

If target owner or table name is defined in the database as anything other than UPPERCASE, be certain to:

  • Type the name in the correct case.
  • Enclose the name in quotation marks, for example "MySchema"."MyTable".
  • To support replication between a source of one database type and a target of another type, the letter case of the names of the source and target columns must be the same, for example the column names on both sides in lower case or both sides in upper case. If the case differs between the source and target column names, use the column mapping feature to map the column names in the configuration file.

Note: This is a basic one-source, one-target configuration using no additional SharePlex configuration features. See Configure SharePlex to replicate data in the SharePlex Administration Guide for important information about creating a configuration file and for additional setup instructions for more complex replication scenarios.

Source configuration example

The following configuration file replicates table HR.Emp from Oracle instance ora112 to target table region1.emp in database mydb on target system sysprod. The source table is case-sensitive.

Datasource:o.ora112

HR."Emp" region1.emp sysprod@r.mydb

Configure SharePlex on the target

  1. Make certain that the database setup meets all of the requirements in Open Target checklist .

  2. Run Database Setup for Teradata (td_setup) to establish a database account and connection information for SharePlex. See Database Setup for Teradata
  3. The following options of the target command are available for use on Teradata targets:

    Characterset category

    Datatype category

    Resources category

    See the SharePlex Reference Guide for more information about the target command.

Set up replication from Oracle to other Open Targets

Overview

SharePlex can connect to a database that supports Open Database Connectivity (ODBC).

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, datatypes and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Review column names

To support replication between a source of one database type and a target of another type, the letter case of the names of the source and target columns must be the same, for example the column names on both sides in lower case or both sides in upper case. If the case differs between the source and target column names, use the column mapping feature to map the column names in the configuration file.

See the SharePlex Administration Guide for more information about column mapping with SharePlex.

Configure SharePlex on the source

Set up SharePlex and the database on the Oracle source system as follows.

Make keys available to SharePlex

To replicate from an Oracle source to an Open Target target, you must make key information available to SharePlex.

Enable Oracle supplemental logging

Enable PK/UK supplemental logging in the Oracle source database. SharePlex must have the Oracle key information to build an appropriate key on the target.

Set SP_OCT_USE_SUPP_KEYS parameter

Set the SP_OCT_USE_SUPP_KEYS parameter to a value of 1. This parameter directs SharePlex to use the columns set by Oracle's supplemental logging as the key columns when a row is updated or deleted. When both supplemental logging and this parameter are set, it ensures that SharePlex can always build a key and that the SharePlex key will match the Oracle key.

See the SharePlex Reference Guide for more information about this parameter.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information.

Datasource:o.SID

src_owner.table

tgt_owner.table

host@r.database_name

where:

  • SID is the Oracle SID of the source Oracle database..
  • src_owner.table is the owner and name of the source table.
  • tgt_owner.table is the owner and name of the target table.*
  • host is the name of the target system.
  • r. identifies the target as non-Oracle.
  • database_name is the name of the target database. Important! database_name must be the actual name of the database, not a data source name (DSN).

* Important! 

If target owner or table name is defined in the database as anything other than UPPERCASE, be certain to:

  • Type the name in the correct case.
  • Enclose the name in quotation marks, for example "MySchema"."MyTable".
  • To support replication between a source of one database type and a target of another type, the letter case of the names of the source and target columns must be the same, for example the column names on both sides in lower case or both sides in upper case. If the case differs between the source and target column names, use the column mapping feature to map the column names in the configuration file.

Note: This is a basic one-source, one-target configuration using no additional SharePlex configuration features. See Configure SharePlex to replicate data in the SharePlex Administration Guide for important information about creating a configuration file and for additional setup instructions for more complex replication scenarios.

Source configuration example

The following configuration file replicates table HR.Emp from Oracle instance ora112 to target table region1.emp in database mydb on target system sysprod. The source table is case-sensitive.

Datasource:o.ora112

HR."Emp" region1.emp sysprod@r.mydb

Configure SharePlex on the target

Configure SharePlex and the database on the target system as follows.

Note: Make certain that the database setup meets all of the requirements in Open Target checklist .

Install the ODBC driver

Make certain to download the correct ODBC driver for your database.

Set up ODBC

  1. Install and test the ODBC driver that is appropriate for your target database. SharePlex provides the OTS utility for testing the ability of SharePlex to connect to and post through ODBC. For more information, see the OTS documentation in the SharePlex Reference Guide.
  2. On the target system, configure ODBC connection information for use by Post to connect to the target database. You have the following options for configuring this connection information:

    • On Windows, create a user or system DSN (Data Source Name) by using Data Sources (ODBC) in the Administrative Tools section of the Windows control panel. See the Windows documentation or your system administrator. If using a DSN, you must set the Post user name and password for the target database with the connection comand. See Set connection information with the connection command.
    • On Unix and Linux, you can do either of the following:

      • Configure a user or system DSN on the target system according to the instructions provided with the database. Test the DSN by using it to connect to the target database. If the connection is successful, copy the ODBC configuration files to the odbc subdirectory of the SharePlex variable-data directory. Set the LD_LIBRARY_PATH environment variable to the location of the database ODBC driver.

        or...

      • Set the ODBC connection information in the Post configuration. See Set connection information with the connection command.

Set connection information with the connection command

Use the connection command to:

  • Set the Post user name and password if you created a DSN.
  • Set all of theODBC connection information if a DSN does not exist.

To set connection information

  1. Create a user account for SharePlex in the target database. This account must be granted the privileges to connect, query the metadata structures of the database, create and update tables in the SharePlex database or schema, and perform full DML and supported DDL operations. Make certain that this user can connect successfully to the database through ODBC outside SharePlex.
  2. Run sp_ctrl.
  3. Execute the connection command with the set option, once for each keyword.

    connection r.database_name set keyword=value

Option1: Input when a DSN exists
Keyword Value to enter

user

The database user assigned to SharePlex

password

The password for the SharePlex user

dsn

The DSN of the database.

IMPORTANT! user, password, and dsn are the only required keywords if a DSN exists.

 

Option 2: Input when a DSN does not exist (Unix and Linux)
Keyword Value to enter

user

The database user assigned to SharePlex

password

The password for the SharePlex user

port

The database port number.

server

The name or IP address of the database server.

driver

The full path to the ODBC driver on the database server.

 

Option 3: Connect string when a DSN does not exist (Unix and Linux)
Keyword Value to enter

user

The database user assigned to SharePlex

password

The password for the SharePlex user

connect_string A user-defined connection string. When using your own connection string, make certain it includes all of the required elements to make a successful ODBC connection, but omit the user name and password. Use the connection command with the user and password options to supply user information.

Connection command examples

DSN exists

connection r.mydb set user=myuser

connection r.mydb set password=mypassword

connection r.mydb set dsn=mydsn

DSN does not exist

connection r.mydb set user=myuser

connection r.mydb set password=mypassword

connection r.mydb set port=1234

connection r.mydb set server=server1

connection r.mydb set driver=/database/ODBC/lib/databasedriver.so

DSN does not exist, use connection string

connection r.mydb set user=myuser

connection r.mydb set password=mypassword

connection r.mydb set connect_string=”driver=/database/ODBC/lib/databasedriver.so;server=server1;port=1234;uid=myuser;pwd=mypassword”

Remove a connection value

Use connection with the reset option to remove SharePlex connection settings.

To remove a specific connection value

connection r.database_name reset keyword

To remove all connection values

connection r.database_name reset

Examples

connection r.mydb reset port

connection r.mydb reset

View connection values

Use connection with the show option to view SharePlex connection settings.

To view connection values for a database

connection r.database_name show

To view connection settings for all local databases

connection show all

Map datatypes

To support the accurate posting of replicated Oracle ALTER TABLE commands to the correct datatypes in an Open Target target, you need to map Oracle datatypes to the datatypes of the target database. Use the target command with the set datatype option to perform this mapping.

Note: For DML operations, SharePlex queries the target database to determine the appropriate datatype mapping for replicated data.

To map datatypes

target r.database [queue queuename ] set datatype src_datatype=tgt_datatype

Where: src_datatype is the Oracle datatype and tgt_datatype is the target datatype to which you are mapping the Oracle datatype.

See the SharePlex Release Notes for mapping options.

For more information about the target command, see the SharePlex Reference Guide.

Set up replication from Oracle to JMS

Overview

The SharePlex Post process can connect and write to a JMS (Java Messaging Service) queue or topic. The data is written as XML records that include the data definitions, the operation type, and the changed column values. This data is written as a sequential series of operations as they occurred on the source, which can then be posted in sequential order to a target database or consumed by an external process or program.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, datatypes and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Configure SharePlex on the source

When replicating data to JMS, configure the source database and SharePlex on the source system as follows.

Enable supplemental logging

On the source system, enable PK/UK supplemental logging in the Oracle source database. SharePlex must have the Oracle key information to build an appropriate key on the target.

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, UNIQUE) COLUMNS;

Set SP_OCT_USE_SUPP_KEYS parameter

On the source system, set the SP_OCT_USE_SUPP_KEYS parameter to a value of 1. This parameter directs SharePlex to use the columns set by Oracle's supplemental logging as the key columns when a row is updated or deleted. When both supplemental logging and this parameter are set, it ensures that SharePlex can always build a key and that the SharePlex key will match the Oracle key.

See the SharePlex Reference Guide for more information about this parameter.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information. The structure that is required in a configuration file varies, depending on your replication strategy, but this shows you the required syntax for routing data through a JMS queue or topic.

Datasource:o.SID
src_owner.table !jms[:tgt_owner.table] host

where:

  • SID is the Oracle SID of the source Oracle database.
  • src_owner.table is the owner and name of the source table.
  • !jms is a required keyword indicating SharePlex is posting to JMS.
  • :tgt_owner.table is optional and specifies the owner and name of the target table. Use if either component is different from that of the source table. Allow no spaces between !jms and :tgt_owner.table. Type case-sensitive names in the correct case and enclose them within double quotes, as in "MySchema"."MyTable"
  • host is the name of the target system.

Note: For more information, see Configure SharePlex to replicate data.

Source configuration example

Datasource:o.ora112

MY_SCHEMA.MY_TABLE !jms:"MySchema2"."MyTable2" sysprod

Configure SharePlex on the target

These instructions establish the Java provider on the target and configure the SharePlex Post process to connect to a JMS structure. SharePlex supports Active MQ and IBM MQ targets.

Configure SharePlex on an Active MQ target

These instructions configure the SharePlex Post process to post to an Active MQ queue or topic.

NoteS:

  • Active MQ does not necessarily need to be installed on the same machine where SharePlex is running, but you will need to copy the ActiveMQ jar files to a location in the SharePlex installation directory according to these instructions.
  • SharePlex is compiled for Java Runtime Engine (JRE) 1.6, but ActiveMQ 5.13 and 5.14 use JRE 1.7. Therefore, these instructions include creating a soft link to point to the JRE 1.7 binary. If the soft link is not created, SharePlex returns an error like the following:

    INFO 2016-05-10 01:43:42.000860 [x.jms+jsd-olinux-01] com.quest.shareplex.openbridge.stomp.internal.Connector - Connector x.jms+jsd-olinux-01 at tcp://localhost:61613 stopped.

    ERROR 2016-05-10 01:43:42.000861 [x.jms+jsd-olinux-01] com.quest.shareplex.openbridge.stomp.internal.Connector - org/apache/activemq/jndi/ActiveMQInitialContextFactory : Unsupported major.minor version 51.0

    java.lang.UnsupportedClassVersionError: org/apache/activemq/jndi/ActiveMQInitialContextFactory : Unsupported major.minor version 51.0

The following directory naming conventions are used in the examples in this procedure:

Name Directory
SharePlex The SharePlex installation directory
amq The working directory for Active MQ within the SharePlex directory.

To configure post to the amq target

  1. Install and configure JRE 1.7 on the SharePlex target server.
  2. Create an amq directory under SharePlex/lib/providers.

    shareplex/lib/providers/amq

  3. Copy the following provider JAR files and HS dependencies to the amq directory.

    shareplex/lib/providers/amq/activemq-all.jar

    shareplex/lib/providers/amq/slf4j.jar

  4. Configure the JAVA_HOME and JRE_HOME variables to point to the JRE 1.7 home directory.
  5. Add the new JAVA_HOME to the PATH variable.
  6. Create a soft link under the SharePlex/.app-modules/openbridge directory to point to the JRE 1.7 binary:

    1. Shut down SharePlex, if running.
    2. Change directories to SharePlex/.app-modules/openbridge.
    3. Issue the following command:

      MV jre/ jre1.6/

    4. Locate and symlink the JRE 1.7 to SharePlex/.app-modules/openbridge/jre
    5. Issue the following command:

      ln –s /home/xxx/java7/jre1.7.0_09 jre

      The following is an example:

      hostname/home/xxx/prod80/.app-modules/openbridge> ls -ltr

      total 24

      drwxr-xr-x 2 xxx dba 4096 Jan 11 14:04 temp/

      drwxr-xr-x 8 xxx dba 4096 Jan 11 14:04 jre1.6/

      drwxr-xr-x 2 xxx dba 4096 Jan 11 14:05 plugins/

      drwxr-xr-x 2 xxx dba 4096 Jan 11 14:05 configuration/

      -r—1 xxx dba 160 Jan 11 14:05 bridge.ini

      -rwxr-xr-x 1 xxx dba 711 Jan 11 14:05 bridge*

      lrwxrwxrwx 1 xxx dba 31 May 10 15:08 jre -> /home/xxx/java7/jre1.7.0_09/

  7. Set all values of producerFlowControl="true" to "false". Additional content about this parameter is available at: http://activemq.apache.org/producer-flow-control.html.
  8. Start sp_cop. (Do not activate the configuration yet.)
  9. Run sp_ctrl.
  10. Issue the target command to configure Post to post to the JMS queue or topic. The following are example commands.

    See View and change JMS settings for command explanations and options.

    If posting to a JMS queue:

    sp_ctrl> target x.jmsset jms factory_class=org.apache.activemq.jndi.ActiveMQInitialContextFactory

    sp_ctrl> target x.jms set jms provider_url=tcp://w2k3-64bit:61616

    sp_ctrl> target x.jms set jms lib_location=shareplex/lib/providers/amq

    sp_ctrl> target x.jms set jms queuename=SHAREPLEX.Q1

    If posting to a JMS topic:

    sp_ctrl> target x.jmsset jms factory_class=org.apache.activemq.jndi.ActiveMQInitialContextFactory

    sp_ctrl> target x.jms set jms provider_url=tcp://w2k3-64bit:61616

    sp_ctrl> target x.jms set jms lib_location=shareplex/lib/providers/amq

    sp_ctrl> target x.jmsset jms destination=topic

    sp_ctrl> target x.jms set jms queuename=SHAREPLEX.Q1

Configure SharePlex on an IBM MQ target

These instructions configure the SharePlex Post process to post to an IBM MQ queue or topic.

The following directory naming conventions are used in the examples in this procedure:

Name Directory
MQClient The IBM MQ client installation directory
SharePlex The SharePlex installation directory
ibmmq The working directory for IBM MQ within the SharePlex directory.

To configure post to the IBM MQ target

  1. Install and configure the latest version of Java Runtime Engine (JRE) on either the database server or an intermediate server where the SharePlex Post process will be running.
  2. Set the JAVA_HOME and JRE_HOME environment variables to the appropriate directories.
  3. Add the JAVA_HOME and JRE_HOME paths to the PATH variable as part of the SharePlex operating system user profile.
  4. Install the IBM MQ client into the MQClient directory on either the database server or an intermediate server where the SharePlex Post process will be running.
  5. Copy the .bindings file that is generated by the IBM MQ Explorer utility from the middle-tier server to the MQClient/lib directory on the SharePlex target server.

    Note: The Post process uses this file to obtain information about the IBM MQ middle-tier server.

  6. If using IBM MQ 8.x, create a soft link to the JRE that ships with the MQ 8.x distribution. (The JRE 1.6. that ships with SharePlex is compatible with MQ 7.x but not 8.x.)

    1. $ cd SharePlex/.app-modules/openbridge
    2. $ mv jre jre.old

    3. $ ln –s /MQClient/java/jre64/jre jre

  7. Create an ibmmq directory within the SharePlex/lib/providers directory.

  8. Copy the .bindings file and all of the .jar files from the MQClient/java/lib directory to the SharePlex/lib/providers/ibmmq directory.

    $ cp MQClient/java/lib/.bindings SharePlex/lib/providers/ibmmq

    $ cp MQClient/java/lib/*.jar SharePlex/lib/providers/ibmmq

  9. Create a soft link to the MQClient/lib directory from the ibmmq directory that you created.

    $ cd SharePlex/lib/providers/ibmmq

    $ ln –sMQClient/libibmmq

  10. Start sp_cop. (Do not activate the configuration yet.)
  11. Run sp_ctrl.
  12. Issue the target command to configure Post to post to the JMS queue or topic. The following are example commands.

    See View and change JMS settings for command explanations and additional options.

    If posting to a JMS queue:

    sp_ctrl> target x.jmsset jms factory_name=SharePlexQcf

    sp_ctrl> target x.jmsset jms lib_location=ibmmq

    sp_ctrl> target x.jms set jms provider_url=file:/MQClient/app/mqm83client/lib

    sp_ctrl> target x.jms set jms queuename=SHAREPLEX.Q1

    sp_ctrl> target x.jms set jms user=myusername

    sp_ctrl> target x.jms set jms password=mypassword

    If posting to a JMS topic:

    sp_ctrl> target x.jmsset jms factory_name=SharePlexQcf

    sp_ctrl> target x.jmsset jms lib_location=ibmmq

    sp_ctrl> target x.jms set jms provider_url=file:/MQClient/app/mqm83client/lib

    sp_ctrl> target x.jms set jms queuename=SHAREPLEX.T1

    sp_ctrl> target x.jms set jms destination=topic

    sp_ctrl> target x.jms set jms user=myusername

    sp_ctrl> target x.jms set jms password=mypassword

View and change JMS settings

To view current property settings for output to JMS, use the following command:

target x.jms show

To change a property setting, use the following command.

target x.jms [queue queuename] set jms property=value

where:

  • queue queuename is the name of a Post queue. Use this option if there are multiple Post processes.
  • property and value are shown in the following table.

Table 1: JMS target properties

Property Input Value Default

factory_class=factory_class

Required Fully qualified class name of the factory class. Sets the JNDI environmental property java.naming.factory.initial to specify the class name of the initial context factory for the provider.

None

provider_url=url

Required RMI URL with no object name component. This sets the JNDI environmental property java.naming.provider.url to specify the location of the registry that is being used as the initial context.

None

lib_location=path

Required Path to the directory where you installed the JAR files.

None

destination={queue | topic}

Messaging domain. Valid values are queue (port-to-port) or topic (publisher-subscriber model).

queue

factory_name=factory_name

Name of a JNDI connection factory lookup. You can specify multiple names with a comma-separated list, for example: (jndi.name1, jndi.name2).

None

user=user

Name of the user that is attaching to JMS. If authentication is not required, omit this and the password option.

None

password=password

Password of the JMS user.

None

queuename=JMS_queuename

Name of the JMS queue or topic.

OpenTarget

persistent={yes | no}

yes logs messages to disk storage as part of send operations.

no prevents logging.

yes
session_transacted ={yes | no}

no directs Post to issue a JMS commit for every replicated message, making each one immediately visible and consumable. This is the default.

yes directs Post to operate in a transactional manner. In this mode, Post issues a JMS commit (to make messages visible and consumable) at intervals based on the following:

  • Issue a commit before the data is read-released from the Post queue, as controlled by the value set for the SP_OPX_READRELEASE_INTERVAL parameter.
  • Issue a commit every JMS write, as controlled by the value set for the SP_OPX_CHECKPOINT_FREQ parameter, until the SP_OPX_READRELEASE_INTERVAL is reached.
no
properties

Use this option if the JMS provider that you are using cannot consume messages that contain the default set of properties supplied by SharePlex. It enables you to remove or add properties. Supply the properties as a comma-delimited list.

  • To add a property, specify it as name=value.
  • To remove a SharePlex property, prefix the name with a dash. For example this string removes two SharePlex properties: -JMSXDeliveryCount,-JMSXGroupSeq.
None
client_id Use this option if the JMS provider that you are using cannot consume messages that contain the default SharePlex client ID. Set this value to the client ID that your provider accepts. None
commit_frequency

Use this option when Post is configured to post to a JMS server in transactional style (issue a JMS commit at intervals, rather than after every message as directed by the session_transacted property of the target command). This parameter specifies the interval between JMS commits. It works in conjunction with the SP_OPX_READRELEASE_INTERVAL parameter. Valid values are 1 to any positive integer.

1500

Set recovery options

If the JMS process aborts suddenly, or if the machine that it is running on aborts, row changes may be written twice to the JMS target. The consumer must manage this by detecting and discarding duplicates.

Every record of every row-change operation in a transaction has the same transaction ID and is also marked with a sequence ID. These attributes are id and msgIdx, respectively, under the txn element in the XML output (see View the XML format).

The transaction ID is the SCN at the time the transaction was committed and the sequence ID is the index of the row change in the transaction. These two values are guaranteed to be the same if they are re-written to the JMS queue in a recovery situation.

If desired, you can configure the target to include additional metadata with every row-change record by using the following command:

target target [queue queuename] set metadata property[, property]

Table 2: Optional JMS metadata properties

Property Description
time The time the operation was applied on the source.
userid The ID of the database user that performed the operation.
trans The ID of the transaction that included the operation.
size The number of operations in the transaction.

Example

target x.jms set metadata time, userid, trans, size

To reset the metadata

target x.jms [queue queuename] reset metadata

To view the metadata

target x.jms [queue queuename] show metadata

Control memory settings

If you configured Post to process large amounts of transactions to JMS through multiple named Post queues, you may need to allocate more memory to the Post JMS bridge. The default settings are 256MB and 512MB.

To increase the JMS bridge memory

Open the openbridge.ini file and set the following parameters in the format shown:

-Xms=512m

-Xmx=1024m

-XX:PermSize=512m

-XX:MaxPermSize=1024m

Configure debugging

You can configure Post to log whether each write to a queue or topic succeeded or failed. This is controlled with the debugging setting.

To enable debugging

Open the openbridge.ini file and set the -OB_debugToLog parameter to true.

-OB_debugToLog=true

View the XML format

To view the XML formatting of the records that SharePlex sends to JMS, see the format category of the target command documentation in the SharePlex Reference Guide.

Set up replication from Oracle to Kafka

Overview

The SharePlex Post process can connect and write to a Kafka broker. The data can be written in JSON or XML output as a sequential series of operations as they occurred on the source, which can then be consumed by a Kafka consumer.

These instructions contain setup instructions that are specific to this target. Install SharePlex on the source and target according to the appropriate directions in this manual before performing these setup steps.

For the versions, datatypes and operations that are supported when using SharePlex to replicate to this target, see the SharePlex Release Notes.

Guidelines for posting to Kafka

  • A SharePlex Post process acts as a Kafka producer. A SharePlex Post process can write to one or more topics that have one or more partitions.
  • The SharePlex Post process does not create a topic itself, but you can configure the Kafka broker to auto-create topics.

Configure SharePlex on the source

When replicating data to Kafka, configure the source database and SharePlex on the source system as follows.

Enable supplemental logging

On the source system, enable PK/UK supplemental logging in the Oracle source database. SharePlex must have the Oracle key information to build an appropriate key on the target.

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY, UNIQUE) COLUMNS;

Set SP_OCT_USE_SUPP_KEYS parameter

On the source system, set the SP_OCT_USE_SUPP_KEYS parameter to a value of 1. This parameter directs SharePlex to use the columns set by Oracle's supplemental logging as the key columns when a row is updated or deleted. When both supplemental logging and this parameter are set, it ensures that SharePlex can always build a key and that the SharePlex key will match the Oracle key.

See the SharePlex Reference Guide for more information about this parameter.

Configure replication

On the source, create a SharePlex configuration file that specifies capture and routing information. The structure that is required in a configuration file varies, depending on your replication strategy, but this shows you the required syntax for routing data to Kafka.

Datasource:o.SID
src_owner.table !kafka[:tgt_owner.table] host

where:

  • SID is the Oracle SID of the source Oracle database.
  • src_owner.table is the owner and name of the source table.
  • !kafka is a required keyword indicating SharePlex is posting to Kafka.
  • :tgt_owner.table is optional and specifies the owner and name of a target table. Use this feature if you want the data to appear as if it came from a table other than the source table. Allow no spaces between !kafka and :tgt_owner.table. Type case-sensitive names in the correct case and enclose them within double quotes, as in "MySchema"."MyTable"
  • host is the name of the target system.

Note: See Configure SharePlex to replicate data.

Source configuration example

Datasource:o.ora112

MY_SCHEMA.MY_TABLE !kafka sysprod

Configure SharePlex on the target

These instructions configure the SharePlex Post process to connect to Kafka. You must have a running Kafka broker.

To configure post to Kafka

  1. Create a Kafka topic.
  2. Start sp_cop. (Do not activate the configuration yet.)
  3. Run sp_ctrl.
  4. Issue the target command to configure posting to a Kafka broker and topic. The following are example commands.

    sp_ctrl> target x.kafkaset kafka broker=host1:9092,host2:9092,host3:9092

    sp_ctrl> target x.kafka set kafka topic=shareplex

    See View and change Kafka settings for command explanations and options.

    Note: Specify more than one broker so that SharePlex will attempt to connect to the other brokers in the list if any one of them is down.

Set the Kafka record format

SharePlex can output to either XML or JSON format as input to Kafka. XML is the default. To set the input format and specify format options, use one of the following target commands:

target x.kafka set format record=json

or:

target x.kafka set format record=xml

 

To view samples of these formats, see the format category of the target command documentation in the SharePlex Reference Guide.

View and change Kafka settings

To view current property settings for output to Kafka, use the following target command:

target x.kafka show

To change a property setting, use the following command.

target x.kafka [queue queuename] set kafka property=value

where:

  • queue queuename is the name of a Post queue. Use this option if there are multiple Post processes.
  • property and value are shown in the following table.

Table 3: Kafka target properties

Property Input Value Default
broker=broker

Required. The host and port number of the Kafka broker, or a comma delimited list of multiple brokers. This list is the bootstrap into the Kafka cluster. So long as Post can connect to one of these brokers, it will discover any other brokers in the cluster.

localhost:9092
client_id=ID

Optional. A user-defined string that Post will send in each request to help trace calls.

None
compression.code={none, gzip, snappy} Optional. Controls whether data is compressed in Kafka. Options are none, gzip or snappy. None
partition={number | rotate | rotate trans}

Required. One of the following:

  • A fixed partition number: Directs Post to post messages only to the specified partition number. For example, setting it to 0 directs Post to post only to partition 0. This option is suitable for use in testing or if the target has multiple channels of data posting to the same Kafka topic.
  • The keyword rotate: Directs Post to apply messages to all of the partitions of a topic in a round-robin fashion. The partition changes with each new message. For example if a topic has three partitions, the messages are posted to partitions 0,1,2,0,1,2, and so on in that order.
  • The keyword rotate trans: This is similar to the rotate option, except that the partition is incremented with each transaction rather than with each message. For example, if a topic has three partitions, the messages are posted to partition 0 until the commit, then to partition 1 until the commit, and so on in that order. This option is suitable if you are replicating multiple tables to a single topic. It allows you to distribute data across several partitions, while still preserving all of the operations of a transaction together in a single partition. This enables a consumer that reads from a single partition to receive a stream of complete transactions.
None
request.required.acks=value Optional. This is a Kafka client parameter. By default it is set to a value of -1, which means all. Consult the Kafka documentation about this subject, because all really means all in-sync replicas. This parameter can be used in conjunction with the min.insync.replicas broker parameter to tune behavior between availability and data consistency. Important: It is possible for data to be lost between a Kafka producer (SharePlex in this case) and a Kafka cluster, depending on these settings. None
topic=topic_name

Required. The name of the target Kafka topic.

This string may contain the special sequences %o or %t. The %o sequence is replaced by the owner name of the table that is being replicated. The %t sequence is replaced by the table name of the table that is being replicated. This feature may be used in conjunction with a Kafka server setting of auto.create.topics.enabled set to 'true'. Also view your server settings for default.replication.factor and num.partitions because these are used as defaults when topics are auto created.

shareplex
threshold_size=packet_size

Optional. The approximate network packet size*, in kilobytes, that Post sends to the Kafka broker.

Notes:

  • Maximum packet size is 128000 KB.
  • Packet size is approximate.
10000KB

* To avoid latency, if Post detects no more incoming messages, it sends the packet to Kafka immediately without waiting for the threshold to be satisfied.

Set recovery options

If the Kafka process aborts suddenly, or if the machine that it is running on aborts, row changes may be written twice to the Kafka topic. The consumer must manage this by detecting and discarding duplicates.

Every record of every row-change operation in a transaction has the same transaction ID and is also marked with a sequence ID. These attributes are id and msgIdx, respectively, under the txn element in the XML output (see ).

The transaction ID is the SCN at the time the transaction was committed, and the sequence ID is the index of the row change in the transaction. These two values are guaranteed to be the same if they are re-written to the Kafka topic in a recovery situation.

If desired, you can configure Post to include additional metadata with every row-change record by using the following command:

target x.kafka [queue queuename] set metadata property[, property]

Table 4: Optional metadata properties

Property Description
time The time the operation was applied on the source.
userid The ID of the database user that performed the operation.
trans The ID of the transaction that included the operation.
size The number of operations in the transaction.

Example

target x.kafka set metadata time, userid, trans, size

To reset the metadata

target x.kafka [queue queuename] reset metadata

To view the metadata

target x.kafka [queue queuename] show metadata

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating