conn_snapshot.sh
conn_snapshot.sh
Use this script to take a fresh copy of an Oracle table for replication.
Shell Script Usage
conn_snapshot.sh -t <TABLE_OWNER.TABLE_NAME> [-f <FILE_TYPE>] [-s <FIELD_SEPARATOR>] [-e <CREATE_EXTERNAL_TABLE>] [-h <HIVE_HOME_DIR>] [-m <NUM_OF_MAPPERS>] [-n <CHANGES_THRESHOLD>] [-i <TIME_THRESHOLD>] [-r] [-d] [-v] [--help] [--version]
Options
| Parameter |
Description |
|
-t <TABLE_OWNER.TABLE_NAME> |
Name and owner of the table to import from Oracle. Required. |
|
-f <FILE_TYPE> |
Applicable to HDFS Near Real Time Replication.
File type for import. [Text|Sequence] (Default = Text. Use -f Sequence for sequence file type.)
-f Sequence |
All data is replicated in the Sequence files on the HDFS. To read/write to the Sequence files you need access to the Writable classes used to generate the Sequence files. See the SharePlex Connector for Hadoop lib/sqoop-records directory and SharePlex Connector for Hadoop lib/sqoop-records.jar.
NOTE: SharePlex Connector for Hadoop does not copy to Hive over HDFS for Sequence file type. | |
|
-s <FIELD_SEPARATOR> |
The separator between each field/column. The separator must be enclosed in single quotes. |
|
-e <CREATE_EXTERNAL_TABLE> |
Copy to Hive. [true|false] (Default = false. Use -e true to enable.)
If true then
- Copy to Hive over HDFS if a copy is taken for HDFS Near Real Time Replication.
- Copy to Hive over HBase if a copy is taken for HBase Real Time Replication.
|
|
-h <HIVE_HOME_DIR> |
Path to the Hive home directory.
If not specified the value of the HIVE_HOME environment variable is used. If this option is not set and the HIVE_HOME environment variable is also not set, this parameter will be set as relative to HADOOP_HOME. |
|
-m <NUM_OF_MAPPERS> |
The number of mappers to be used. |
|
-n <CHANGES_THRESHOLD> |
Use to override the default setting for how often SharePlex Connector for Hadoop replicates the table (measured by the number of changes to the table).
NOTES:
- Applicable to HDFS Near Real Time Replication.
- The default setting was set in conn_setup.sh.
- Replication is executed on the first condition met: on the given number of changes to the table or the set time period -i, whichever comes first.
- SharePlex Connector for Hadoop remembers this setting and makes use of it during further execution of conn_snapshot.
|
|
-i <TIME_THRESHOLD> |
Use to override the default setting for how often SharePlex Connector for Hadoop replicates the table (measured by the number of minutes).
NOTES:
- Applicable to HDFS Near Real Time Replication.
- The default setting was set in conn_setup.sh.
- Replication is executed on the first condition met: on the given number of changes to the table -n or the set time period, whichever comes first.
- SharePlex Connector for Hadoop remembers this setting and makes use of it during further execution of conn_snapshot.
|
|
-r -d |
Use to override the settings in conn_setup.sh. If not specified, replicate this Oracle table using HBASE and/or HDFS as per the settings in conn_setup.sh.
|
-r |
A copy of the table is taken for HBase Real Time Replication.
Do not replicate this Oracle table using HDFS.
This overrides the settings in conn_setup.sh. |
|
-d |
A copy of the table is taken for HDFS Near Real Time Replication.
Do not replicate this Oracle table using HBASE.
This overrides the settings in conn_setup.sh. |
|
-r -d |
A copy of the table is taken for HBase Real Time Replication and HDFS Near Real Time Replication.
This overrides the settings in conn_setup.sh. |
NOTE: SharePlex Connector for Hadoop remembers these settings and makes use of them during further execution of conn_snapshot. |
|
-v |
Verbose - Show detailed information of each step. |
|
--help |
Show this help and exit. |
|
--version |
Show version information and exit. |
Example
[user@host bin]$ ./conn_snapshot.sh -t Schema.Table -s ';'
Use Cases
|
Take a copy of the Oracle table Shema.Table for replication over HDFS and / or HBase as per the settings in conn_setup.sh |
conn_snapshot.sh -t Schema.Table -s ';' |
|
As above and ...
- Copy to Hive over HDFS if a copy is taken for HDFS Near Real Time Replication.
- Copy to Hive over HBase if a copy is taken for HBase Real Time Replication.
|
conn_snapshot.sh -t Schema.Table -s ';' -e true |
|
Take a copy of the Oracle table Shema.Table for replication over HBase. Do not replicate over HDFS. |
conn_snapshot.sh -t Schema.Table -s ';' -r |
|
Take a copy of the Oracle table Shema.Table for replication over HDFS. Do not replicate over HBase. Replication is set for every 20 minutes or 100 changes - whichever comes first. |
conn_snapshot.sh -t Schema.Table -s ';' –d –i 20 –n 100 |
conn_ctrl.sh
conn_ctrl.sh
Use this script to start / restart / shutdown SharePlex Connector for Hadoop and reload SharePlex Connector for Hadoop configurations.
Shell Script Usage
conn_ctrl.sh [start|restart|shutdown|reload-conf|--help|--version]
Options
| Parameter |
Description |
|
start |
Start SharePlex Connector for Hadoop. |
|
restart |
Stop and start SharePlex Connector for Hadoop. |
|
shutdown |
Stop SharePlex Connector for Hadoop. |
|
reload-conf |
Reload some of the SharePlex Connector for Hadoop configurations without restarting SharePlex Connector for Hadoop. These configurations include
- HDFS Near Real Time Replication changes and time interval.
|
|
--help |
Show this help and exit. |
|
--version |
Show version information and exit. |
Example
[user@host bin]$ ./conn_ctrl.sh
conn_monitor.sh
conn_monitor.sh
Use this script to show a list of the tables being replicated. Monitor the status of a table under replication.
Shell Script Usage
conn_monitor.sh [--summary|--status <TABLE_OWNER.TABLE_NAME> [--hdfs|--hbase]|--help|--version]
Options
| Parameter |
Description |
|
--summary |
Show a summary of the tables under replication. |
|
--status <TABLE_OWNER.TABLE_NAME> [--hdfs|--hbase] |
Show the status of the given table.
Include the --hdfs option (optional) to limit the display to HDFS Near Real Time Replication.
Include the --hbase option (optional) to limit the display to HBase Real Time Replication. |
|
--help |
Show this help and exit. |
|
--version |
Show version information and exit. |
conn_cleanup.sh
conn_cleanup.sh
Use this script to delete all data (HDFS, HBase and Hive) from a table under replication by SharePlex Connector for Hadoop.
Shell Script Usage
conn_cleanup.sh -t <TABLE_OWNER.TABLE_NAME> [-h <HIVE_HOME_DIR>] [--help] [--version]
Options
| Parameter |
Description |
|
-t <TABLE_OWNER.TABLE_NAME> |
Name and owner of the table data to cleanup. |
|
-h <HIVE_HOME_DIR> |
Path to the Hive home directory.
If not specified the value of the HIVE_HOME environment variable is used. If this option is not set and HIVE_HOME environment variable is also not set, this parameter will be set as relative to HADOOP_HOME. |
|
--help |
Show this help and exit. |
|
--version |
Show version information and exit. |
Example
[user@host bin]$ ./conn_cleanup.sh -t Schema.Table
