Execute the following SharePlex for Oracle command. SharePlex Connector for Hadoop will use the information from this command to ensure the integrity of the data inside of checking for duplicate and missing messages.
sp_ctrl ()> target x.jms set metadata size
TIP: You can confirm the setting by running the target x.jms show command which will show size as a metadata parameter set on the queue. Refer to the SharePlex for Oracle documentation for more information.
JMS (ActiveMQ) parameters can be tuned to improve the overall performance of SharePlex Connector for Hadoop. Results may vary depending on the further tuning and actual availability of resources.
Following are some of the parameters in “ACTIVEMQ_HOME/conf/activemq.xml” that can be tuned to get better performance:
Increase “queue memory limit” and turn off “producer flow control”.
Example: <policyEntry queue=">" producerFlowControl="false" memoryLimit="512mb">
Note: The value of the “queue memory limit” parameter should be less than the “memory usage limit” parameter.
Increase “memory usage limit” of the broker.
Example: <memoryUsage limit="1 gb"/>
Note: Take into consideration the available system memory while setting these parameters. The Memory Usage Limit parameter should not exceed the max memory allowed for JMS (ActiveMQ) process. You can change the Max memory allowed for JMS ActiveMQ by altering the “–Xmx” parameter in “activemq-admin” or “activemq” scripts under “ACTIVEMQ_HOME/bin” with which ActiveMQ gets started (look for ACTIVEMQ_OPTS or ACTIVEMQ_OPTS_MEMORY).
If default KahaDB is getting used as a persistent store, then setting enableJournalDiskSyncs to “false” can result in reduction in time while sending messages from SharePlex Poster to ActiveMQ JMS queue.
Example: <persistenceAdapter> <kahaDB directory="${activemq.data}/kahadb" enableJournalDiskSyncs="false"/> </persistenceAdapter>
If you intend to replicate tables in Hive over HBase then complete the following additional setup steps.
IMPORTANT NOTE: When using connector on IBM BI Hadoop and CDH 5 distributions, jar files specified in this scenario must be appended to HIVE_AUX_JARS_PATH in HIVE_CONF_DIR/hive-env.sh file.
Ensure the Zookeeper Quorum and client port are configured in $HIVE_HOME/conf/hive-site.xml.
<property>
<name>hbase.zookeeper.quorum</name>
<value> ---- PLEASE SPECIFY ---- </value>
<description>A comma separated list (with no spaces) of the IP addresses of all ZooKeeper servers in the cluster.</description>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value> ---- PLEASE SPECIFY ---- </value>
<description>The Zookeeper client port. Default clientPort is 2181.</description>
</property>
Use --auxpath while entering into Hive
hive --auxpath <HIVE_HOME>/lib/hive-hbase-handler-<version>.jar,<HBASE_ HOME>/hbase.jar,<HBASE_HOME>/lib/zookeeper.jar,<HBASE_HOME>/lib/guava-<version>.jar
When using CDH4 or CDH 5 with Kerberos authentication
Export the HIVE_OPTS environment variable with Kerberos parameters as shown below to replicate Hive over HBase.
export HIVE_OPTS="-hiveconf hbase.security.authentication=kerberos -hiveconf hbase.rpc.engine=org.apache.hadoop.hbase.ipc.SecureRpcEngine -hiveconf hbase.master.kerberos.principal=hbase/_HOST@KerberosRelm -hiveconf hbase.regionserver.kerberos.principal=hbase/_HOST@KerberosRelm -hiveconf hbase.zookeeper.quorum=zookeeperQuorum"
TIP: The Kerberos utility kinit allows you to identify yourself to the Kerberos server. “kinit” needs to be invoked (only once) if you are starting a new session.
© 2024 Quest Software Inc. ALL RIGHTS RESERVED. 이용 약관 개인정보 보호정책 Cookie Preference Center