After server crash, export queue not moving (4258925)

A server where SharePlex was running crashed.  After restarting the server, applications, database and SharePlex , export queue status shows as running but the export queue is is not draining. Checking the connection between source and target does not show any issues with the network.

Event_logs shows the following messages:
Source:
12/18/07 19:26 Notice: (Read) Queue write recovery started, sqrw_srcseq=0x194b1069ef50  msg.mpseq=0x194b101ebd50 [sp_ordr(que)/4933]
12/18/07 19:26 Notice: Queue write recovery complete, 4929 duplicate messages skipped , 6581267 bytes total [sp_ordr(que)/4933]
12/18/07 19:26 Notice: Oracle env - ocwar:/oracle/app/product/9.2.0 [sp_ordr(pdb)/4933]

Target:
12/18/07 19:23 System call error: Connection timed out failed to read from socket in import [sp_mport/29048]
12/18/07 19:23 Process exited sp_mport  (from db1 queue queuename) [pid = 29048] - killed due to SIGPIPE
12/18/07 19:43 Process launched: sp_mport (from db1 queue queuename) [pid = 15851]
12/18/07 19:43 Connected to export on db1
12/18/07 19:43 Notice: Queue write recovery started, qrw_srcseq=0x194b01f17f61  msg.mpseq=0x194b01d14561 [sp_mport(que)/15851]
12/18/07 19:43 Notice: Queue write recovery started, qrw_srcseq=0x194b01f18f01  msg.mpseq=0x194b01d164a1 [sp_mport(que)/15851]
12/18/07 19:52 Cannot connect to export on db1
12/18/07 19:52 Process exited sp_mport  (from db1 queue queuename) [pid = 15851] - exit(1)

Server crash caused queue corruption.

Resolution 1

Source:
1. Shutdown Shareplex

2. cd PRODDIR/bin

3. ./qview -i
   qview> fixup all   ==> this can run a few minutes depending on the size of your queue

4. ./qview -i
   qview> qsetup <port>
   qview> open x r  ==> choose the export queue which has the issue
   qview> oread 0 10
  
If the command returns valid data then fixup was succesfull, SharePlex may be restarted and export should be moving. 

If the output from oread shows "NO DATA" then continue with the next step.

5. qview> seekback
   qview> uscan 0 1 3 ==> This will show the first 3 good messages, use this information to release to the first good message.
   qview> rrls <sque #> <qseq #> 
  
CAUTION: Read releasing data will cause Out of sync. Run compare afterwards and resync with compare/repair, export/import or your own resync procedure.

For example

   qview> seekback
   qview> uscan 0 1 3

Sque 0, mid 67761380, qseq 41263563106, 10/03/08 18:46:55, 5064/28050664, transid 3, Insert, AAALwZAAcAAA4MRAAc  "NGDOWNSP"."EVENT_LOG", SCN  8646193704, forward = 1

Sque 0, mid 67761381, qseq 41263563683, 10/03/08 18:46:55, 5064/28052024, transid 3, Insert, AAALwUAAbAAA4REAAP  "NGDOWNSP"."EVENT_LOG", SCN  8646193704, forward = 1

Sque 0, mid 67761382, qseq 41263564260, 10/03/08 18:46:55, 5064/28053368, transid 3, Insert, AAALwjAAeAAA4JAAAj  "NGDOWNSP"."EVENT_LOG", SCN  8646193704, forward = 1

   qview> rrls 0 41263563106

6. Exit qview and start SharePlex

Resolution 2

If the resution 1 doesn't work, it indicates the export queue is completedly corrupted. Resetting of the queue is needed.

On the source

   qview -i

   qview>qsetup <port>

   qview>reset  X

   qview>exit

On the target

   shutdown shareplex

   qview -i

   qview>qsetup <port>

   qview>reset p

   qview>exit

On the target
   sqlplus> truncate table shareplex_trans

Start up shareplex on the source and target

 
CAUTION: Resetting of the queue will erase all messages in the queue and will cause Out of sync. Run compare afterwards and resync with compare/repair, export/import or your own resync procedure.